Bringing Nunchaku 4-bit Diffusion Inference to Diffusers

Hugging Face Blog42d ago

Modern diffusion transformers have unlocked unprecedented capabilities in image, video, and audio generation. However, this progress comes with a significant hardware tax: running these models in standard BF16 precision often demands between 20 GB and 30 GB of VRAM. For the vast majority of users operating on consumer-grade hardware, these models have remained frustratingly out of reach.

While the Diffusers library has already made strides by integrating quantization backends like bitsandbytes, GGUF, torchao, and Quanto, these solutions are primarily weight-only. By storing weights in low precision and dequantizing them on the fly, they successfully reduce memory footprints, but they rarely improve inference speed and can occasionally introduce latency overheads.

Enter SVDQuant, the core technology behind the Nunchaku inference engine. By utilizing 4-bit weights and activations (W4A4), SVDQuant doesn't just shrink the model—it accelerates the denoising loop. Today, we are excited to announce that Nunchaku is now natively integrated into Diffusers.

---

The Evolution of Nunchaku Lite

Previously, utilizing Nunchaku-optimized checkpoints required a specialized, separate inference library. With this update, loading a Nunchaku checkpoint is as seamless as calling from_pretrained(), with no local CUDA compilation required. Furthermore, the new diffuse-compressor toolkit empowers users to quantize their own architectures and share them as standard Diffusers repositories.

Getting Started

To begin, ensure you have the latest versions of the required libraries:

pip install -U diffusers transformers accelerate kernels bitsandbytes

Loading a pre-quantized pipeline is straightforward:

import torch
from diffusers import ErnieImagePipeline

pipe = ErnieImagePipeline.from_pretrained(
    "lite-infer/ERNIE-Image-Turbo-nunchaku-lite-nvfp4_r32-bnb4-text-encoder",
    torch_dtype=torch.bfloat16,
).to("cuda")

image = pipe(
    prompt="A cinematic portrait of a red fox in a misty forest at sunrise, detailed fur, volumetric light",
    height=1024, width=1024, num_inference_steps=8, guidance_scale=1.0,
    generator=torch.Generator("cuda").manual_seed(42),
).images[0]
image.save("output.png")

This implementation requires no custom pipeline classes. The NVFP4 kernels are fetched automatically from the Hugging Face Hub upon first use. On an RTX 5090, this setup generates a 1024x1024 image in approximately 1.7 seconds while consuming only 12 GB of VRAM, a massive improvement over the 24 GB typically required for a BF16 pipeline.

---

Understanding SVDQuant and Nunchaku

Standard 4-bit quantization often fails with diffusion transformers because both weights and activations are prone to significant outliers. SVDQuant solves this by shifting activation outliers into the weights. It represents the most complex segments of a weight matrix using a small 16-bit low-rank branch, while quantizing the remaining residuals to 4 bits.

Nunchaku optimizes this process by using fused kernels for both the 4-bit path and the low-rank branch. By fusing the low-rank down-projection with the quantization kernel and the up-projection with the 4-bit compute kernel, the engine eliminates the memory access bottlenecks typically associated with 16-bit branches.

Introducing Nunchaku Lite

The original Nunchaku engine relies on model-specific fused execution paths (like fused QKV projections). While highly efficient, these are tied to specific architectures, making them difficult to scale. Nunchaku Lite is the new, flexible integration path within Diffusers.

Native Compatibility: It patches nn.Linear modules with runtime SVDQ/AWQ layers before the checkpoint loads.
Kernel Versatility: It utilizes svdq_w4a4 (for attention and MLP projections) and awq_w4a16 (for adaptive normalization and modulation) to balance speed and precision.
Performance: While it doesn't match the extreme speed of the original engine's hard-coded kernels, it provides a consistent 30% speedup alongside significant memory savings.

---

Seamless Integration and Hardware Support

If you have worked with bitsandbytes or torchao, the workflow will feel intuitive. A Nunchaku Lite repository is a standard Diffusers repository, distinguished only by a quantization_config block in the transformer/config.json file.

Hardware Compatibility Matrix

Note: Volta and Hopper architectures are currently unsupported. The system includes built-in validation to prevent hardware-mismatch errors.

---

Benchmarks and Optimization

Nunchaku Lite shines when paired with existing Diffusers optimizations. By leveraging torch.compile, users can push performance even further.

Performance Metrics (RTX PRO 6000, 1024x1024)

| Configuration | Full Pipeline | Denoise Loop | Peak VRAM | Speedup | | :--- | :--- | :--- | :--- | :--- | | BF16 Baseline | 3.00 s | 2.86 s | 31.1 GB | 1.0x | | Nunchaku Lite NVFP4 | 2.27 s | 2.13 s | 20.6 GB | 1.35x | | + torch.compile | 1.68 s | 1.53 s | 20.6 GB | 1.8x | | + NF4 Text Encoder | 2.29 s | 2.13 s | 16.0 GB | 1.35x |

"Nunchaku reduces peak VRAM by up to 50% while simultaneously improving latency by roughly 30%. When combined with torch.compile, we see a total speedup of 1.8x over the BF16 baseline."

---

Quantizing Your Own Models

The diffuse-compressor toolkit provides a comprehensive workflow—calibrate, quantize, package, and publish—for any compatible architecture.

1. Inspection: Use the generic scanner to identify which layers are suitable for SVDQ W4A4 or AWQ W4A16 quantization. 2. Quantization: Execute the quantization script, choosing between int4 or nvfp4 depending on your target hardware. 3. Packaging: Combine the quantized transformer with the pipeline components, ensuring the nunchaku_lite configuration is correctly injected. 4. Verification: Load the model, test the output, and push it to the Hugging Face Hub for the community to enjoy.

A Note on Structural Rewrites

While the generic path handles most models, some architectures (like FLUX.1-dev) require structural rewrites. The original Nunchaku engine fuses separate Q, K, and V projections into a single module. Because the generic path cannot infer these custom groupings, developers can use model-specific target configs and runtime adapters to achieve these advanced optimizations.

---

Conclusion

Nunchaku’s SVDQuant kernels represent a major milestone in making high-end diffusion models accessible to the broader community. By lowering the barrier to entry for both inference and model development, we are enabling a new wave of efficient, high-quality generative AI.

We encourage you to explore the available checkpoints, such as the ERNIE-Image-Turbo variants or the Krea 2 Turbo models, and start experimenting with your own architectures.

Further Resources

Special thanks to the Diffusers maintainers for their guidance, the MIT HAN Lab / Nunchaku team for their foundational research, and the community members who provided invaluable feedback during the development of this feature.

Read Original Source