Claude Code Local LLM Speed by Hardware Tier: What Backends Actually Claim
- 8 min read
Every local-LLM setup guide, including my own, tells you which model to pull for your hardware. Almost none of them tell you what speed to actually expect once Claude Code’s agentic loop is hammering the backend with tool calls, not a single chat prompt. I haven’t run a controlled benchmark rig across every tier myself, so what follows is each backend’s own published throughput claims, organized by hardware tier, with the caveats that matter for agentic use specifically. Treat these as a starting expectation to validate on your own hardware, not a guarantee.
16 GB Unified Memory (M-series MacBook, base tier)
At this tier, the ceiling isn’t the backend, it’s the model. Ollama runs qwen2.5-coder:14b and gemma3:12b comfortably; oMLX’s own docs claim its tiered cache lets a 16 GB machine run models that would otherwise need closer to 24 GB, at some cost to speed on cache misses. Anything meaningfully above ~14B parameters starts swapping to oMLX’s SSD tier or fails outright on plain Ollama.
24 GB Unified Memory / VRAM (mid-tier)
Ollama’s own numbers put qwen3-coder (30B, 3B active) as a comfortable fit at this tier. Rapid-MLX’s project page claims 4.2x faster inference than Ollama on Apple Silicon and a 0.08s cached time-to-first-token — figures from their own benchmarks, not independently verified here. If that claim holds on your hardware, its 17 tool-call parser formats are the more relevant advantage for Claude Code specifically: cached time-to-first-token matters more across a session of repeated tool calls than raw tokens/sec on a single prompt.
NVIDIA GPU, 16 GB VRAM (ExLlamaV3 + TabbyAPI)
ExLlamaV3’s EXL3 quantization format is documented as running Llama-3.1-70B at 1.6 bits-per-weight in under 16 GB VRAM — a real capability, per the project’s own benchmarks. In practice, a well-quantized 32B model (Qwen2.5-Coder-32B, for one) is likely to round-trip through Claude Code’s loop faster than a maximally-quantized 70B one, since at 1.6 bpw you’re trading model size for speed, not getting both.
What Actually Slows Down the Agentic Loop
A backend’s own benchmark number rarely matches what you see inside Claude Code, for three reasons:
Tool-call parsing overhead. Every tool call round-trips through a parser before the model sees the result. Backends that ship dedicated tool-call formats (Rapid-MLX’s 17 parser formats, for one) cut this overhead measurably versus a generic OpenAI-compatible bridge like LiteLLM.
Context re-processing. Claude Code re-sends accumulated context on every turn. Below 64K context support, you’ll see the model re-truncating and losing earlier instructions, which shows up as repeated or drifting behavior, not just slower output.
Cache misses on long sessions. oMLX’s two-tier cache is the exception, but on Ollama and Rapid-MLX, a long session with a lot of file reads will start evicting earlier context, and every cache miss costs a full re-read at un-cached speed.
The Practical Takeaway
If you’re on 16 GB unified memory, don’t fight the tier — Gemma 3 12B or Qwen2.5-Coder:14b on Ollama is the realistic starting point. At 24 GB, Rapid-MLX’s claimed throughput advantage is worth testing against Ollama on your own hardware before you commit to it. On NVIDIA hardware, a well-quantized 32B model is the safer default over a maximally-quantized 70B one for anything Claude Code is doing turn-by-turn.
None of this changes the guidance in the 2026 setup guide on which backend to install. It just tells you what to actually validate before you assume a benchmark claim holds for agentic, tool-call-heavy use.
Related Reading
Sizing local AI infrastructure for your team's hardware? Let's talk through the tradeoffs.
Schedule a Call
