September 22, 2026
GGUF models will stop bloating memory: Transformers loads llama.cpp quants
On September 22, Hugging Face added loading of llama.cpp GGUF quants to Transformers without dequantization. Qwen3.5-4B in Q4_K_M takes 2,74 GB instead of 8,42 GB in BF16.
Without a compatible `ggml-quantization` kernel, Transformers dequantized GGUF at load time and used more memory. Qwen3.5 can now be loaded through `from_pretrained` with a GGUF file, and Hugging Face recommends starting with Q4_K_M.
How to install. The first route is designed for Macs with Apple Silicon, the two latest PyTorch releases, `transformers` from `main` and the `kernels` package. The loaded model can be served to a local OpenAI-compatible client through `transformers serve`.
The packed route currently works through MPS, while padding and batching still need work.
