NVIDIA Reports Up to 73% Faster JAX Model Training on Blackwell GPUs Using NVFP4 Format

NVIDIA has disclosed performance improvements for large language model training using a new low-precision numerical format called NVFP4 on its latest Blackwell architecture GPUs. The announcement, based on experiments with Google's JAX framework and the MaxText training library, reflects the industry's ongoing effort to reduce the cost and time required for artificial intelligence model training.

According to a developer blog post, NVIDIA achieved speedups ranging from 1.31× to 1.73× over an FP8 baseline when training the Llama 3 8B model on Blackwell GPUs using the NVFP4 format. This represents up to 73% performance improvement, and the company reported that these gains came with no measurable accuracy loss. The company said it maintained a similar training loss curve across 10,000 pretraining steps.

Balancing Numerical Precision and Training Efficiency

Numerical precision in AI model training involves a balance between computational speed, memory usage, and final model quality. Traditionally, 32-bit floating-point (FP32) format was standard, but in recent years the industry has moved toward 16-bit (FP16), Brain Float 16 (BF16), and 8-bit (FP8) formats. Each step reduced precision in exchange for higher computational throughput and lower memory bandwidth requirements.

NVFP4 extends this trend with a 4-bit floating-point format. Theoretically, a 4-bit format can halve memory usage and increase throughput compared with 8-bit formats. However, in practice, the representable numerical range and precision are limited, which can create numerical instability or convergence issues during training.

NVIDIA's results are notable because they suggest NVFP4 can work in real large language model training without accuracy loss, despite these theoretical concerns. The company reported that similar training loss curves were maintained over 10,000 pretraining steps, indicating that the model learned in a pattern comparable to FP8.

The Role of Blackwell Architecture

These performance gains are closely tied to the hardware design of Blackwell GPUs. Blackwell is NVIDIA's latest datacenter GPU architecture, incorporating dedicated hardware accelerators for low-precision arithmetic. The NVFP4 format is designed to use these hardware capabilities, combining software optimization with hardware support.

MaxText is a JAX-based high-performance training library developed by Google, providing implementations for large language model training. NVIDIA's emphasis on MaxText integration highlights collaboration within the JAX ecosystem and suggests that Blackwell's capabilities can be leveraged across frameworks beyond PyTorch or TensorFlow.

Industry Context and Competitive Landscape

This announcement is part of a broader industry effort to reduce AI training costs. Large language model training can require substantial computing expense, with training times ranging from weeks to months. A 73% speedup has the potential to reduce these costs and timelines, making large-scale model training more accessible to more organizations.

Competitors are moving in similar directions. AMD is developing its own low-precision formats, Google's TPUs are optimized around Brain Float formats, and Intel and other new entrants are seeking positions in the AI accelerator market. NVIDIA's NVFP4 announcement can be viewed in the context of this competitive environment.

Practical Considerations and Constraints

However, applying these results to production environments involves several considerations. First, NVIDIA's disclosed results are based on a specific model (Llama 3 8B) and specific training configuration (MaxText recipe). Whether similar results will occur with different model architectures, datasets, or training hyperparameters requires additional validation.

Second, 10,000 pretraining steps may represent only a portion of the complete training process. Large models undergo hundreds of thousands to millions of training steps, and numerical errors could accumulate over extended periods. It is not clear whether NVIDIA has confirmed the same accuracy maintenance over longer training runs.

Third, NVFP4 is a format specific to the Blackwell architecture, so leveraging it requires upgrading to the latest hardware. Organizations using existing Hopper or Ampere generation GPUs cannot immediately benefit from these capabilities.

Future Outlook

Advances in low-precision training are important as AI model scale and complexity continue to increase. The industry is already discussing models with trillions of parameters, and the computing resources required to train such models continue to grow. Technologies like NVFP4 can help moderate this growth and enable more efficient training.

Additionally, low-precision formats can play an important role in the inference stage. When deploying trained models to production environments, lower precision can mean faster response times and lower operational costs. If the same low-precision format can be used for both training and inference, the efficiency of the entire AI pipeline may improve.

NVIDIA's announcement shows how collaboration among hardware manufacturers, framework developers, and model researchers can lead to practical performance improvements. How quickly the JAX and MaxText communities adopt NVFP4, and whether similar results can be reproduced with other models and tasks, will help determine the long-term impact of this technology.

The adoption of low-precision formats also has economic and environmental implications. Reduced training time can lower power consumption, which may decrease both datacenter operating costs and carbon emissions. As the AI industry faces sustainability pressure, efficient training technologies offer a way to address both environmental and economic considerations.