What hardware runs Mistral Nemo 12B?
Mistral Nemo 12B is a 12.25B-parameter dense model (apache-2.0 license). The base under half the roleplay finetunes ever made. Natural prose, light guardrails, happiest at 16K.
All figures assume an f16 KV cache, a 0.6 GB display reserve on the GPU, and 64 GB of DDR5 system RAM for the offload tiers. Tune these in the calculator.
Minimum VRAM by quant and context
| Quant | File size | 8K | 16K | 32K | 64K | 128K |
|---|---|---|---|---|---|---|
| Q2_K * | 4.5 GB | 7.6 GB | 8.9 GB | 11.5 GB | 16.8 GB | 27.3 GB |
| Q3_K_M * | 5.7 GB | 8.8 GB | 10.1 GB | 12.7 GB | 18 GB | 28.5 GB |
| Q4_K_M | 7.0 GB | 10.1 GB | 11.4 GB | 14 GB | 19.3 GB | 29.8 GB |
| Q5_K_M | 8.1 GB | 11.2 GB | 12.6 GB | 15.2 GB | 20.4 GB | 30.9 GB |
| Q6_K | 9.4 GB | 12.5 GB | 13.8 GB | 16.4 GB | 21.7 GB | 32.2 GB |
| Q8_0 | 12.1 GB | 15.2 GB | 16.6 GB | 19.2 GB | 24.4 GB | 34.9 GB |
| IQ4_XS | 6.3 GB | 9.4 GB | 10.7 GB | 13.3 GB | 18.6 GB | 29.1 GB |
Full-GPU figures: weights + f16 KV cache + overhead. * below our recommended floor of Q4_K_M.
GPU compatibility
| GPU | 8K | 16K | 32K | 64K | 128K |
|---|---|---|---|---|---|
| NVIDIA GeForce RTX 5090 | Q8_0 | Q8_0 | Q8_0 | Q8_0 | Q5_K_M |
| NVIDIA GeForce RTX 5080 | Q8_0 | Q6_K | Q5_K_M | IQ4_XS | Q8_0 |
| NVIDIA GeForce RTX 5070 Ti | Q8_0 | Q6_K | Q5_K_M | IQ4_XS | Q8_0 |
| NVIDIA GeForce RTX 5070 | Q5_K_M | Q4_K_M | IQ4_XS | Q8_0 | Q8_0 |
| NVIDIA GeForce RTX 5060 Ti 16GB | Q8_0 | Q6_K | Q5_K_M | IQ4_XS | Q8_0 |
| NVIDIA GeForce RTX 5060 | IQ4_XS | IQ4_XS | IQ4_XS | Q8_0 | Q8_0 |
| NVIDIA GeForce RTX 4090 | Q8_0 | Q8_0 | Q8_0 | Q6_K | Q4_K_M |
| NVIDIA GeForce RTX 4080 SUPER | Q8_0 | Q6_K | Q5_K_M | IQ4_XS | Q8_0 |
| NVIDIA GeForce RTX 4070 Ti SUPER | Q8_0 | Q6_K | Q5_K_M | IQ4_XS | Q8_0 |
| NVIDIA GeForce RTX 4070 | Q5_K_M | Q4_K_M | IQ4_XS | Q8_0 | Q8_0 |
| NVIDIA GeForce RTX 4060 Ti 16GB | Q8_0 | Q6_K | Q5_K_M | IQ4_XS | Q8_0 |
| NVIDIA GeForce RTX 4060 | IQ4_XS | IQ4_XS | IQ4_XS | Q8_0 | Q8_0 |
| NVIDIA GeForce RTX 3090 | Q8_0 | Q8_0 | Q8_0 | Q6_K | Q4_K_M |
| NVIDIA GeForce RTX 3080 10GB | IQ4_XS | IQ4_XS | IQ4_XS | Q8_0 | Q8_0 |
| NVIDIA GeForce RTX 3070 | IQ4_XS | IQ4_XS | IQ4_XS | Q8_0 | Q8_0 |
| NVIDIA GeForce RTX 3060 Ti | IQ4_XS | IQ4_XS | IQ4_XS | Q8_0 | Q8_0 |
| NVIDIA GeForce RTX 3060 12GB | Q5_K_M | Q4_K_M | IQ4_XS | Q8_0 | Q8_0 |
| NVIDIA GeForce RTX 2080 Ti | Q4_K_M | IQ4_XS | IQ4_XS | Q8_0 | Q8_0 |
| NVIDIA GeForce GTX 1080 Ti | Q4_K_M | IQ4_XS | IQ4_XS | Q8_0 | Q8_0 |
| AMD Radeon RX 9070 XT | Q8_0 | Q6_K | Q5_K_M | IQ4_XS | Q8_0 |
| AMD Radeon RX 7900 XTX | Q8_0 | Q8_0 | Q8_0 | Q6_K | Q4_K_M |
| AMD Radeon RX 7900 XT | Q8_0 | Q8_0 | Q8_0 | Q4_K_M | Q8_0 |
| AMD Radeon RX 7800 XT | Q8_0 | Q6_K | Q5_K_M | IQ4_XS | Q8_0 |
| AMD Radeon RX 7600 XT | Q8_0 | Q6_K | Q5_K_M | IQ4_XS | Q8_0 |
| AMD Radeon RX 6800 XT | Q8_0 | Q6_K | Q5_K_M | IQ4_XS | Q8_0 |
| AMD Radeon RX 6700 XT | Q5_K_M | Q4_K_M | IQ4_XS | Q8_0 | Q8_0 |
| Intel Arc B580 | Q5_K_M | Q4_K_M | IQ4_XS | Q8_0 | Q8_0 |
| Intel Arc A770 16GB | Q8_0 | Q6_K | Q5_K_M | IQ4_XS | Q8_0 |
| Apple M2 (16GB) | Q5_K_M | Q4_K_M | IQ4_XS | Q8_0 | Q8_0 |
| Apple M2 Pro (32GB) | Q8_0 | Q8_0 | Q8_0 | Q6_K | Q4_K_M |
| Apple M2 Max (64GB) | Q8_0 | Q8_0 | Q8_0 | Q8_0 | Q8_0 |
| Apple M2 Ultra (128GB) | Q8_0 | Q8_0 | Q8_0 | Q8_0 | Q8_0 |
| Apple M3 (16GB) | Q5_K_M | Q4_K_M | IQ4_XS | Q8_0 | Q8_0 |
| Apple M3 Pro (36GB) | Q8_0 | Q8_0 | Q8_0 | Q8_0 | IQ4_XS |
| Apple M3 Max (64GB) | Q8_0 | Q8_0 | Q8_0 | Q8_0 | Q8_0 |
| Apple M3 Ultra (96GB) | Q8_0 | Q8_0 | Q8_0 | Q8_0 | Q8_0 |
| Apple M4 (16GB) | Q5_K_M | Q4_K_M | IQ4_XS | Q8_0 | Q8_0 |
| Apple M4 Pro (48GB) | Q8_0 | Q8_0 | Q8_0 | Q8_0 | Q8_0 |
| Apple M4 Max (64GB) | Q8_0 | Q8_0 | Q8_0 | Q8_0 | Q8_0 |
| CPU only / integrated graphics | Q8_0 | Q8_0 | Q8_0 | Q8_0 | Q8_0 |
Fits on GPUExpert offloadPartial offloadCPU only
Quant guidance
Our floor for Mistral Nemo 12B is Q4_K_M — below that, quality degrades faster than the VRAM savings are worth. Prefer the highest quant that still lands "Fits on GPU" at your context length in the table above.
Recommended run command
Q4_K_M at 32K on a NVIDIA GeForce RTX 5080-class GPU (Fits on GPU):
llama-server -m Mistral-Nemo-Instruct-2407-Q4_K_M.gguf -c 32768 --flash-attn -ngl 99
Frequently asked questions
How much VRAM does Mistral Nemo 12B need?
At Q4_K_M and 32K context, Mistral Nemo 12B needs about 14 GB of VRAM to run fully on GPU (weights + KV cache + overhead).
What is the smallest GPU that can run Mistral Nemo 12B?
The NVIDIA GeForce RTX 5080 (16 GB) is the smallest GPU in our set that runs Mistral Nemo 12B well at 32K context.
What quantization should I use for Mistral Nemo 12B?
We recommend Q4_K_M or higher. Q4_K_M weighs 7.0 GB (4.88 bits/weight); going below Q4_K_M costs noticeable quality on a model this size.
How long a context can Mistral Nemo 12B handle?
Mistral Nemo 12B supports up to 128K tokens. KV cache grows linearly with context.
Can I run Mistral Nemo 12B without a GPU?
Yes, at reduced speed: on CPU with 64 GB of DDR5 it manages roughly 4 tokens/sec at 8K context (Q8_0).