What hardware runs Llama 3.3 70B?
Llama 3.3 70B is a 70.55B-parameter dense model (llama3.3 license). Still the dense 70B reference. Prose depth the MoEs haven't matched; the hardware bill is the price of admission.
All figures assume an f16 KV cache, a 0.6 GB display reserve on the GPU, and 64 GB of DDR5 system RAM for the offload tiers. Tune these in the calculator.
Minimum VRAM by quant and context
| Quant | File size | 8K | 16K | 32K | 64K | 128K |
|---|---|---|---|---|---|---|
| Q2_K * | 24.6 GB | 28.9 GB | 31.5 GB | 36.6 GB | 46.9 GB | 67.4 GB |
| Q3_K_M | 31.9 GB | 36.3 GB | 38.8 GB | 44 GB | 54.2 GB | 74.7 GB |
| Q4_K_M | 39.6 GB | 44 GB | 46.5 GB | 51.7 GB | 61.9 GB | 82.4 GB |
| Q5_K_M | 46.5 GB | 50.9 GB | 53.4 GB | 58.6 GB | 68.8 GB | 89.3 GB |
| Q6_K | 53.9 GB | 58.3 GB | 60.8 GB | 66 GB | 76.2 GB | 96.7 GB |
| Q8_0 | 69.8 GB | 74.2 GB | 76.8 GB | 81.9 GB | 92.1 GB | 112.6 GB |
| IQ4_XS | 35.3 GB | 39.7 GB | 42.3 GB | 47.4 GB | 57.6 GB | 78.1 GB |
Full-GPU figures: weights + f16 KV cache + overhead. * below our recommended floor of Q3_K_M.
GPU compatibility
Fits on GPUExpert offloadPartial offloadCPU only
Quant guidance
Our floor for Llama 3.3 70B is Q3_K_M — below that, quality degrades faster than the VRAM savings are worth (large models tolerate low bpw better than small ones, which is why the floor is lower here). Prefer the highest quant that still lands "Fits on GPU" at your context length in the table above.
Recommended run command
Q3_K_M at 32K on a Apple M2 Max (64GB)-class GPU (Fits on GPU):
llama-server -m Llama-3.3-70B-Instruct-Q3_K_M.gguf -c 32768 --flash-attn -ngl 99
Frequently asked questions
How much VRAM does Llama 3.3 70B need?
At Q3_K_M and 32K context, Llama 3.3 70B needs about 44 GB of VRAM to run fully on GPU (weights + KV cache + overhead).
What is the smallest GPU that can run Llama 3.3 70B?
The Apple M2 Max (64GB) (48 GB) is the smallest GPU in our set that runs Llama 3.3 70B well at 32K context.
What quantization should I use for Llama 3.3 70B?
We recommend Q3_K_M or higher. Q4_K_M weighs 39.6 GB (4.82 bits/weight); going below Q3_K_M costs noticeable quality on a model this size.
How long a context can Llama 3.3 70B handle?
Llama 3.3 70B supports up to 128K tokens. KV cache grows linearly with context.
Can I run Llama 3.3 70B without a GPU?
Yes, at reduced speed: on CPU with 64 GB of DDR5 it manages roughly 1 tokens/sec at 8K context (Q5_K_M).