llama.cpp's MoE Expert Offload: --n-cpu-moe and Streaming, Explained

If you’re running a mixture-of-experts model locally - Qwen3-Coder, DeepSeek-class models - the whole appeal is that only a fraction of the parameters activate per token. Qwen3-Coder’s 30B model only has 3B active at a time. The problem is that llama.cpp still has to keep every expert somewhere, active or not, and “somewhere” is the thing that’s constrained on a 16GB or 24GB box. This is what the offload flags and the newer streaming work actually do about that.

—n-cpu-moe: the flag that exists today

--n-cpu-moe N keeps the MoE feed-forward expert weights for the first N layers in system RAM instead of VRAM. Higher N means less VRAM used and more CPU work per token; lower N means more experts stay on the GPU and generation is faster. It’s a knob, not an optimization - you’re trading speed for headroom, layer by layer, until the model fits.

In practice this is how a 30B-class MoE model becomes usable on a 16GB card: attention and the active-per-token compute stay on the GPU, and enough of the expert weight sits in system RAM to keep the model from failing to load at all. The tradeoff shows up directly in tokens/sec, so the right value of N is whatever value stops the model from OOMing, not a value below that.

What’s changing: streaming experts from disk

There’s an open feature request and multiple community implementations - at least five independent approaches as of this writing - for streaming routed experts straight from disk, on demand, instead of pre-loading them into RAM at all. The mechanism: the router finishes computing which experts a token needs before attention finishes computing on the current layer. That gap is a prefetch window. The next layer’s needed experts can be fetched from disk while the current layer is still computing, and one project’s discussion cites a roughly 2x throughput improvement from overlapping that I/O with compute this way, versus loading everything into RAM up front.

This matters for the same reason --n-cpu-moe matters, but it changes the ceiling: instead of “how much RAM do I have to hold experts in,” the question becomes “how fast is my disk, and how good is the prefetch.” A model bigger than your RAM becomes runnable at all, not just something you avoid.

It isn’t merged and stable everywhere yet. Treat it as something to watch on a memory-constrained NVIDIA or Strix Halo box - not something to plan a hardware purchase around today, since the exact throughput you’ll see depends on implementation and disk speed in ways the current published numbers don’t fully pin down.

Where this actually matters for Claude Code

Claude Code’s agentic loop is tool-call-heavy and re-sends accumulated context every turn, which is a different access pattern than a single long-form generation. --n-cpu-moe tuning still applies the same way - find the lowest N that doesn’t OOM - but the re-processing on every turn means a cache miss costs you a full re-read at whatever speed the CPU-offloaded experts run at. If you’re on a MoE model and seeing inconsistent turn-to-turn latency, that’s usually the offload boundary, not the model itself.

Once expert streaming stabilizes, it’s the more relevant development for anyone running these models on the 16GB or 24GB tiers.

---

Related Reading

• Claude Code Local LLM Speed by Hardware Tier
• Running Claude Code with a Local LLM in 2026: No Proxy Required