Training a large language model with numbers stored in just four bits sounds like a recipe for a worse model, yet it is one of the most promising ways to make pretraining cheaper. A new study from researchers at NVIDIA, the Institute of Science and Technology Austria (ISTA) and Red Hat AI asks a practical question: of all the tricks proposed for 4-bit training, which ones are really needed? The paper, “What Matters for NVFP4 Training?”, was posted to arXiv on October 17, 2026. It is a workshop paper, accepted at the ICML 2026 Workshop on Resource-Adaptive Foundation Model Inference, so it has had lighter review than a main-conference paper. Treat its numbers as promising evidence rather than settled fact.

The problem: cheaper math, noisier learning

Most of the compute in pretraining goes into large matrix multiplications. Every step down in numerical precision, from 16-bit to 8-bit and now to 4-bit, lets hardware do more of those multiplications per second and per watt. NVFP4 is a 4-bit floating-point format with block scaling that is supported natively on NVIDIA’s Blackwell GPUs. The catch is noise. Rounding every weight, activation and gradient to one of a handful of values introduces error, and the paper notes that in the 4-bit regime this noise can affect convergence even when individual operations look accurate.

Several research groups have published recipes to tame that noise, and they do not agree on what is essential. One recipe quantizes weights in two dimensions so the forward and backward passes see consistent values; another prefers one-dimensional quantization because it pairs naturally with random rotations in the backward pass. The authors argue that the right answer is hard to infer from short experiments, because many accuracy effects only appear late in training or at larger scale.

What the researchers did

Rather than invent a new method from scratch, the team put the main ingredients from earlier recipes into one common training stack and compared them under identical conditions. They tested two model types from NVIDIA’s Nemotron family: a dense 8-billion-parameter hybrid Mamba-Transformer (Nemotron-H 8B) and a 30B-A3B mixture-of-experts model (Nemotron 3 Nano), trained for up to one trillion tokens. They tracked the “loss gap”, the relative difference in training loss compared with the same model trained in BF16, a standard 16-bit format. They also checked downstream benchmarks covering knowledge, reasoning, math and coding.

The design axes they varied include: whether some layers stay in higher precision; how the scale for each block of numbers is chosen; whether weights are quantized in one or two dimensions; and how bias in the gradient estimate is handled. Bias matters because small systematic errors in gradients can accumulate over a long run, whereas random errors tend to average out.

Main findings

The authors report three headline results from their ablation studies.

First, every technique in the combined recipe measurably improved the loss trajectory. No single trick did all the work, which is why they call their final recipe “Mosaic NVFP4”: it blends the best pieces of earlier approaches, including random Hadamard transforms on both backward matrix multiplications, an unbiased gradient estimator called MS-EDEN introduced in the Quartet II work, and an adaptive scale-selection method called Four Over Six.

Second, keeping selected layers in higher precision is necessary for full accuracy recovery at this scale. In all the reported full runs the last 15% of the network stayed in BF16. Earlier work had shown that close variants of the recipe could converge on small models over short runs with every layer in 4-bit, but at this scale the authors saw instabilities without the high-precision layers. They read this as a representation failure that error-reduction tricks cannot fix.

Third, not all tensors in the backward pass benefit equally from removing bias, which leaves room for complementary techniques on the tensors where de-biasing helps less.

On the numbers: the dense 8B model trained with Mosaic NVFP4 finished within a 1.2% relative training-loss gap of BF16, improving the previous best gap by 0.24 percentage points. The mixture-of-experts model finished within 0.81%, improving on the previous recipe by 0.36 percentage points. The authors say these differences exceed the run-to-run variability they measured across three repeated runs at the 1T-token horizon, and they describe the result as the strongest FP4 training outcome at this scale to date, to the best of their knowledge.

A methodological warning

Perhaps the most useful lesson is about evaluation. The rankings of recipes did not settle until well past 150 billion tokens, and weaker recipes often looked better than the strongest ones early on. The authors suggest this can come from random differences in the sign vectors used by the rotations, or from noise in training. Their conclusion is that short ablations are good for catching catastrophic instability but are not enough to decide whether a new method preserves accuracy over a full run. Anyone choosing a low-precision recipe on the basis of a 10-billion-token test should read that with caution.

Limits to keep in mind

The paper itself is candid about scope. It covers medium-scale dense and MoE language models and NVFP4 only. It does not establish that the findings hold for larger production runs, other modalities, post-training, or other 4-bit formats. The 1.2% and 0.81% gaps are measured in training loss, and a small loss gap does not guarantee identical behavior on every downstream task. Several authors are affiliated with NVIDIA, which sells the hardware the format targets, and the work was done on its models, so independent replication would add confidence. The speed and energy savings that motivate the work are not the subject of this study; it measures accuracy recovery, not throughput.

What to watch next

  • Whether the Mosaic recipe is reproduced by groups outside NVIDIA and ISTA on other architectures.
  • Results at larger model sizes and longer token budgets, where the authors say conclusions are untested.
  • Published throughput and energy measurements for full 4-bit training runs.
  • Whether open frameworks adopt the recipe as a default option for Blackwell-class hardware.
  • Studies of 4-bit methods during post-training and reinforcement learning stages.

Key takeaways

  • NVFP4 training stores numbers in four bits to cut compute cost, but needs careful recipes to match 16-bit quality.
  • The study combines techniques from several prior recipes and reports loss gaps of 1.2% (8B dense) and 0.81% (30B-A3B MoE) versus BF16 after up to 1T tokens.
  • Selective high-precision layers, here the last 15% of the network, were needed at this scale.
  • Short ablations can mislead: recipe rankings only stabilized after more than 150B tokens.
  • The paper is a workshop preprint with limited scope, so independent replication is still needed.

Frequently asked questions

Is this paper peer-reviewed?

It was accepted at an ICML 2026 workshop, which involves some review but is generally lighter than a main conference. It is also available as an arXiv preprint.

Does 4-bit training make models worse?

In this study the trained models end up very close to BF16 in training loss, within roughly 1% on the tested setups, but a loss gap is not the same as identical downstream quality. See our explainer on quantization for the inference-time version of this trade-off.

Why does this matter for AI costs?

Pretraining is a large share of the expense described in our piece on the economics of training and inference. If 4-bit training can match quality, the same hardware could train larger models or finish runs sooner, which also bears on the hardware dynamics in why Nvidia dominates AI.