What hardware runs GPT OSS 120B?
GPT OSS 120B is a 116.83B-parameter mixture-of-experts model with 5.71B active parameters (apache-2.0 license). The big filtered brain. Expert offload on a 24 GB card plus 64 GB RAM makes it a realistic home deployment.
All figures assume an f16 KV cache, a 0.6 GB display reserve on the GPU, and 64 GB of DDR5 system RAM for the offload tiers. Tune these in the calculator.
Minimum VRAM by quant and context
| Quant | File size | 8K | 16K | 32K | 64K | 128K |
|---|---|---|---|---|---|---|
| Q2_K * | 58.3 GB | 60.7 GB | 61.3 GB | 62.4 GB | 64.8 GB | 69.6 GB |
| Q3_K_M | 58.3 GB | 60.7 GB | 61.3 GB | 62.5 GB | 64.9 GB | 69.6 GB |
| Q4_K_M | 58.5 GB | 60.9 GB | 61.4 GB | 62.6 GB | 65 GB | 69.8 GB |
| Q5_K_M | 58.6 GB | 61 GB | 61.6 GB | 62.7 GB | 65.1 GB | 69.9 GB |
| Q6_K | 58.9 GB | 61.3 GB | 61.9 GB | 63.1 GB | 65.5 GB | 70.2 GB |
| Q8_0 | 59.0 GB | 61.4 GB | 62 GB | 63.2 GB | 65.6 GB | 70.3 GB |
Full-GPU figures: weights + f16 KV cache + overhead. * below our recommended floor of Q3_K_M.
GPU compatibility
Fits on GPUExpert offloadPartial offloadCPU only
Quant guidance
Our floor for GPT OSS 120B is Q3_K_M — below that, quality degrades faster than the VRAM savings are worth (large models tolerate low bpw better than small ones, which is why the floor is lower here). Prefer the highest quant that still lands "Fits on GPU" or “Expert offload” at your context length in the table above.
Recommended run command
Q3_K_M at 32K on a NVIDIA GeForce RTX 5060-class GPU (Expert offload):
llama-server -m gpt-oss-120b-Q3_K_M.gguf -c 32768 --flash-attn -ngl 99 --n-cpu-moe 36
Frequently asked questions
How much VRAM does GPT OSS 120B need?
At Q3_K_M and 32K context, GPT OSS 120B needs about 62.5 GB of VRAM to run fully on GPU (weights + KV cache + overhead). As a mixture-of-experts model it can also run with far less VRAM via expert offload, keeping experts in system RAM.
What is the smallest GPU that can run GPT OSS 120B?
The NVIDIA GeForce RTX 5060 (8 GB) is the smallest GPU in our set that runs GPT OSS 120B well at 32K context, using expert offload with 64 GB of system RAM.
What quantization should I use for GPT OSS 120B?
We recommend Q3_K_M or higher. Q4_K_M weighs 58.5 GB (4.3 bits/weight); going below Q3_K_M costs noticeable quality on a model this size.
How long a context can GPT OSS 120B handle?
GPT OSS 120B supports up to 128K tokens. KV cache grows linearly with context.
Can I run GPT OSS 120B without a GPU?
Not realistically — even at 8K context the weights don't fit in 64 GB of system RAM.