Best local LLM for rag & documents on a Apple M2 Ultra (128GB) (2026)
All figures assume an f16 KV cache, a 0.6 GB display reserve on the GPU, and 64 GB of DDR5 system RAM for the offload tiers. Tune these in the calculator.
The verdict
Qwen3.5 35B A3B Q8_0
Fits on GPU at 16K · RAG & documents score 9/10 · ≈ 87 tok/s
The meta pick, full stop. Near-dense-30B quality at 3B-active speed, and expert offload puts it on 8 GB cards.
llama-server -m Qwen3.5-35B-A3B-Q8_0.gguf -c 16384 --flash-attn -ngl 99
Worthy alternates
Qwen3.5 122B A10B Q5_K_M
Fits on GPU · ≈ 44 tok/s · RAG & documents 9/10
Arguably the best model a 128 GB Mac can run today. The unified-memory sweet spot the 397B can't reach.
Qwen3.5 9B Q8_0
Fits on GPU · ≈ 55 tok/s · RAG & documents 8/10
The new small default. Frontier-distilled, natively multimodal, and embarrassingly good for 6 GB of weights.
Tune this for your exact RAM and settings in the calculator → · All models on the Apple M2 Ultra (128GB)
Frequently asked questions
What is the best local LLM for rag & documents on a Apple M2 Ultra (128GB)?
Qwen3.5 35B A3B at Q8_0 — it scores 9/10 for rag & documents and runs as "Fits on GPU" at 16K context on the Apple M2 Ultra (128GB).
How much context do I need for rag & documents?
We recommend 24K tokens for rag & documents (minimum 12K). These picks are computed at 16K.
How fast will it run on a Apple M2 Ultra (128GB)?
Roughly 87 tokens/sec for Qwen3.5 35B A3B — fast for interactive use.
Do I need more than 96 GB of VRAM for rag & documents?
No — the pick above needs 37.4 GB of VRAM at 16K.
What settings should I use?
Start with our command: llama-server -m Qwen3.5-35B-A3B-Q8_0.gguf -c 16384 --flash-attn -ngl 99 — then tune context and KV quant in the fit calculator.