Researchers at New York University and the University of Calgary posted a paper on November 7, 2026, showing that a language model can learn a new task without any of its weights being given new values. Their method, Deletion Tuning, only decides which existing weights to set to zero. Zeroing fewer than 0.05% of all weights matched full fine-tuning on the largest models tested, from 7B to 30B parameters, and the resulting file for an 8B model is 8.4 MB instead of roughly 30.6 GB. The arXiv listing marks the paper as a NeurIPS 2026 paper, so it has been through conference peer review, but the results have not yet been reproduced by other groups.
What Deletion Tuning changes in a model
Standard fine-tuning can move every parameter. LoRA, the most common lightweight alternative, freezes the model and trains a small low-rank add-on. Deletion Tuning is stricter than both. The base model stays frozen, and the method learns one yes-or-no decision per weight: keep the pretrained value, or replace it with zero.
The decisions apply to a single matrix in each transformer block, the “gate” projection inside the feed-forward layer. During training, each weight gets a trainable score that passes through a smooth on/off function, which is gradually sharpened until it behaves like a hard switch.
The scores start from Wanda, an existing pruning heuristic that estimates which weights are safe to remove. With the default setting, about 0.18% of gate-projection weights are zeroed before training begins, and training then decides the final pattern with no fixed sparsity target.
Pruning normally removes weights to make a model smaller or faster while keeping what it already knows. Here the same operation is used to teach. A control experiment supports that reading: Wanda pruning on its own, at the best of eight sparsity levels, stayed within 2.4 points of the untouched base model on all six backbones, while trained masks gained up to 53.2 points.
Results across three kinds of post-training
The paper tests the method in the three settings that make up most of the post-training stack that turns a base model into an assistant.
Reinforcement learning. Using GRPO, a form of reinforcement learning with verifiable rewards, the authors trained Llama models on GSM8K math problems and Qwen2.5 models on a math corpus evaluated across six held-out benchmarks. Accuracy in percent, full fine-tuning (FFT) against Deletion Tuning:
- Llama 8B, GSM8K: base 60.27, FFT 83.24, Deletion Tuning 83.55.
- Qwen 7B, six-benchmark mean: base 40.18, FFT 51.89, Deletion Tuning 52.32.
- Qwen 14B: base 44.37, FFT 54.87, Deletion Tuning 55.43.
- Qwen3-30B-A3B (mixture of experts): base 49.41, FFT 58.43, Deletion Tuning 58.54.
Smaller models tell a different story. On Llama 1B, Deletion Tuning reached 54.44 against 60.50 for FFT, and on Qwen 0.5B it scored 17.64 against 19.01. The authors say the gap closes as models grow.
Supervised fine-tuning. On Llama 3B, Deletion Tuning scored 22.28 on MMLU-Pro subjects (FFT 21.19) and tied FFT at 11.39 on the BBEH reasoning suite. On Llama 1B it trailed on BBEH, 5.65 against 8.35.
On-policy distillation. With Qwen 7B, it reached 69.07 on the ToolAlpaca tool-use task (FFT 68.04) and 70.61 on a chemistry question set (FFT 71.20).
Less forgetting of earlier skills
Fine-tuning a model on one task often erodes others. The authors tracked six unrelated benchmarks, including MMLU, HellaSwag and HumanEval, while training Llama models on GSM8K. Full fine-tuning ended 1.3 to 3.1 points below the base model on that suite. LoRA and a random sparse baseline called MTH lost up to 5.4 points. Deletion Tuning stayed within one point of the base model throughout.
The random sparse baseline is a useful comparison because it edits about as many weights (0.05% of parameters) as Deletion Tuning deletes (0.035% to 0.046% at these sizes) yet forgot more. The authors take this as evidence that the act of deleting, and not just the small number of edits, limits the damage.
How it compares with LoRA in practice
Readers of our practical overview of fine-tuning with LoRA will ask what this adds. On accuracy and size, the paper’s own tables show less than the headline suggests. On Llama 8B, a rank-1 LoRA scored 83.38 with a 10.0 MB adapter; Deletion Tuning scored 83.55 at 8.4 MB.
A worked example with the paper’s figures makes the point. A team keeping 100 task-specific versions of an 8B model would store about 3 TB of full checkpoints, about 1 GB of rank-1 LoRA adapters, or about 840 MB of deletion masks. The step from full fine-tuning to either adapter is large; the step from LoRA to masks is not.
The clearer difference is retention. In the paper’s tests LoRA drifted from the base model’s behavior and lost prior-task accuracy, while masks did not. The trade-off runs the other way during training: Deletion Tuning keeps one trainable score per masked weight, so the authors note it needs less memory than full fine-tuning but more than LoRA. The paper also does not claim faster inference, since so few weights are removed.
Limits of the evidence
The reported scores are best-checkpoint validation accuracies, a choice applied to every method. Forgetting was measured only for Llama models under reinforcement learning. Supervised fine-tuning was tested only up to 3B parameters, and the reinforcement learning tasks are all math. The authors list continual learning over a sequence of tasks as an untested next step, although low forgetting is the main argument for it.
What to watch next
- Independent replications, especially on coding and agent tasks outside math.
- A real continual-learning test in which several masks are learned one after another.
- Whether combining masks for different tasks merges skills, an idea the authors raise but do not test.
- Support for mask adapters in common fine-tuning libraries.
Key takeaways
- Deletion Tuning adapts a model only by zeroing selected pretrained weights; no weight receives a new value.
- Fewer than 0.05% of all weights are deleted, and results match full fine-tuning at 7B to 30B scale.
- Below about 3B parameters the method trails full fine-tuning on several tasks.
- Masks are 3,600 to 4,700 times smaller than full checkpoints under reinforcement learning, but only slightly smaller than a rank-1 LoRA.
- Prior-task accuracy stayed within one point of the base model, while other methods lost up to 5.4 points.
Frequently asked questions
What is Deletion Tuning?
It is a fine-tuning method that freezes a language model and learns a binary mask deciding which weights to set to zero. The mask is the entire adapter.
Is Deletion Tuning better than LoRA?
In the paper it matched LoRA on accuracy at 7B and 8B scale with a slightly smaller file and forgot much less of the model’s earlier abilities. It uses more memory than LoRA during training.
Does deleting weights make the model faster?
The paper does not claim a speedup. Fewer than 0.05% of weights are zeroed in an unstructured pattern, which is too few to cut inference cost.
