Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original

In the current landscape of artificial intelligence, the pursuit of efficiency is often a zero-sum game. The standard industry playbook for deploying Large Language Models (LLMs) typically involves a two-stage degradation process: first, structural compression—pruning layers, heads, or neurons—followed by quantization, which reduces weight precision to 4 bits to slash memory and compute requirements. While this pipeline is essential for fitting models onto consumer-grade hardware, it almost invariably results in a "capability tax." Reasoning, mathematical prowess, and coding accuracy often suffer, necessitating a "healing" phase to recover lost performance before a model is deemed production-ready.

Recent high-profile open-weight releases, including the gpt-oss series, NVIDIA’s Nemotron family, and our own Hypernova 60B, have all leaned on various iterations of this compress-then-heal methodology. However, a fundamental question has remained largely unaddressed by the research community: once a model has undergone both structural compression and quantization, how effective is the recovery process, and what is the optimal strategy to execute it?

Our latest research, Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs, tackles this head-on. By applying our novel Quantization-Aware Healing (QAH) technique to a GPT-OSS 120B model—compressed to 60B parameters and quantized to MXFP4—we have achieved a breakthrough: a model that outperforms its own full-precision (bfloat16) counterpart across seven out of nine major benchmarks. This result effectively inverts the traditional hierarchy, proving that a 4-bit model can be smaller, more cost-effective, and more accurate than the 16-bit checkpoint from which it was derived.

The Limitations of Conventional Healing

To understand why QAH represents a paradigm shift, one must first examine the shortcomings of existing healing pipelines. Most current methods follow a rigid sequence: compress, quantize, then heal. The primary differentiator between these methods lies in the final stage.

The industry standard is Quantization-Aware Training (QAT). This approach injects "fake-quantization" operators into the forward pass, continuing fine-tuning on a task-specific loss to help weights adapt to low-precision constraints. While effective, QAT is notoriously expensive. It requires re-running complex multi-stage post-training—including supervised fine-tuning, RLHF, and agentic tuning—through a noisy, low-precision forward pass. Furthermore, our findings suggest that QAT is prone to instability; if training persists beyond the optimal point, the model’s performance can degrade rapidly.

An alternative, Quantization-Aware Distillation (QAD), attempts to bypass the heavy lifting of re-training by distilling a frozen full-precision teacher into the quantized student using a KL-divergence loss. This works seamlessly when only quantization is involved, as a perfect full-precision version of the model exists to serve as a teacher. However, when structural compression is introduced—changing the actual architecture of the model—this assumption collapses. Because there is no independently trained full-precision version of the smaller architecture, the only available teacher is the recovered bfloat16 checkpoint. This creates a "ceiling effect," where the student is anchored to a degraded target, limiting its potential accuracy to that of the recovered checkpoint.

The QAH Advantage: Architecture-Agnostic Distillation

Our Quantization-Aware Healing (QAH) approach removes this performance ceiling by changing the teacher-student relationship. Instead of distilling from the recovered, structurally compressed checkpoint, QAH distills directly from the original, pre-compression model.

"The student never sees hard labels, only the teacher's output distribution, matched through KL divergence on the logits. This reframes what the quantization stage is doing. Under QAH, it is no longer a lossy post-processing step; it is a second, full pass of distillation against the original teacher."

Because the teacher’s output distribution is architecture-agnostic, the mismatch in size and shape between the 120B teacher and the 60B student becomes irrelevant. The student is not merely compensating for information lost to quantization; it is actively absorbing knowledge that the earlier recovery stage lacked the time or data to transfer.

Key Benefits of the QAH Framework:

• Superior Accuracy: By leveraging the original, full-size teacher, the student gains access to higher-quality signals, leading to performance gains in complex reasoning and math.
• Training Stability: KL distillation ties the student to a fixed teacher distribution. Once the student aligns with the teacher, there is no pressure for it to drift, unlike cross-entropy task loss, which continues to push the student toward hard labels indefinitely.
• Long-Context Efficiency: To handle long-context healing (up to 32k tokens), we utilize a memory-efficient chunked KL-divergence loss. This computes the KL divergence slice-by-slice, avoiding the need to materialize the full vocabulary-by-sequence grid, thus fitting within a fixed GPU memory budget.

Benchmarking the Results

We applied QAH to a GPT-OSS 120B model, compressing it to 60B parameters and recovering it in bfloat16 before re-quantizing to MXFP4. When compared against the best available 60B bfloat16 checkpoint, the QAH-trained model demonstrated significant improvements:

| Benchmark | 60B BF16 (Recovered) | 60B MXFP4 (QAH) | QAH vs BF16 | | :--- | :--- | :--- | :--- | | AA-LCR (Long-context reasoning) | 35.3 | 42.7 | +7.4 | | AIME 2025 (Math) | 70.7 | 76.3 | +5.6 | | Aider (Agentic coding) | 38.2 | 40.9 | +2.7 | | τ²-bench (Tool use) | 59.4 | 61.7 | +2.3 | | GPQA Diamond (Science) | 65.7 | 67.4 | +1.7 | | IFBench (Instruction following) | 58.4 | 59.9 | +1.5 | | LiveCodeBench (Coding) | 65.5 | 66.5 | +1.0 |

The results are striking. The largest gains occur precisely where compression typically causes the most damage: long-context reasoning and mathematics. Even more impressively, the 4-bit QAH model managed to surpass the full-size 120B teacher on LiveCodeBench (66.5 vs. 66.0) and came within a narrow margin of the teacher on GPQA Diamond.

QAH vs. QAT: A Head-to-Head Comparison

To isolate the impact of our loss function, we conducted a head-to-head comparison against QAT by quantizing a GPT-OSS 9B model to MXFP4.

• Speed: QAH reached its peak performance in approximately 100 steps, roughly 7 times faster than the 700 steps required by QAT.
• Stability: Once QAH reached its peak, it remained stable for the duration of training. In contrast, QAT exhibited a sharp collapse, shedding nearly 19 points of performance by step 1,200.

This creates a significant practical advantage for deployment. A QAT-trained model requires rigorous early stopping to avoid shipping a degraded checkpoint. A QAH-trained model, however, is inherently more robust, as the KL distillation objective provides no incentive for the model to deviate from the teacher’s distribution once convergence is achieved.

Changing the Deployment Paradigm

The implications of this research extend beyond mere benchmark scores. By utilizing QAH, the 4-bit model consumes roughly four times less weight memory than its bfloat16 counterpart and halves the compute requirements per token compared to the 120B teacher. For organizations currently shipping models in bfloat16, the combined parameter and precision reduction could result in an eight-fold decrease in compute costs.

Ultimately, this research suggests that quantization should no longer be viewed as a "tax" paid for efficiency. Instead, it is an opportunity to further refine and teach the model. As part of Multiverse Computing’s ongoing mission to optimize large-scale AI, QAH provides a robust, efficient, and highly effective path toward making powerful models more accessible, cheaper to run, and more capable than ever before.

For those interested in the technical implementation, including the chunked KL-divergence machinery and distributed training configurations, we invite you to review our full paper or reach out to our team to discuss applying these methods to your own model deployment pipelines.