October 21, 2026
--moe-cache-mib in llama.cpp: GPU expert cache speeds up MoE model 2.2x on RTX 5090

llama.cpp has had a GPU expert cache since Oct 7: Qwen3.8-Flash-Next in Q4_0, weighing 93.7 GiB, runs at 67.8 tokens per second on an RTX 5090, up from 30.8. The cache keeps frequently called experts in VRAM, while rarely used ones are loaded over PCIe.