Nemotron 3 Super 120B needs 2-4 data-center GPUs to run, depending on precision. At full BF16, the model requires ~270GB of VRAM across 4x 80GB cards. Popular VRAM calculators disagree widely on this, so this guide uses NVIDIA's own published deployment requirements as the source of truth, then walks through a vLLM setup and the real cost of running the model.
Takeaways
- Nemotron 3 Super 120B requires 2-4 data-center GPUs. BF16 needs 4x 80GB cards (320GB total); FP8 fits on 2x 80GB with a 32K context cap; NVFP4 runs on a single 96GB Blackwell card.
- All 120B parameters must be loaded into VRAM even though only ~12B activate per token. Hardware planning must start from the full parameter count, not the active count.
- NVFP4 is Blackwell-only. On an H100 (Hopper) or A100 (Ampere), use FP8 or a 4-bit GGUF quant instead.
- Renting beats buying for most teams. A 3x RTX PRO 6000 self-host rig costs around $60,000 built out; 4x A100 on Thunder Compute runs $5.96 with no upfront spend.
- vLLM with tensor parallelism is the standard deployment path. Set
--tensor-parallel-sizeto match your GPU count and--trust-remote-codefor NVIDIA's custom checkpoint.
What Makes Nemotron 3 Super 120B Different to Run
Nemotron 3 Super 120B is a Mixture-of-Experts (MoE) model with 120B total parameters and ~12B active per token. NVIDIA built it on a hybrid LatentMoE architecture interleaving Mamba-2 and attention layers, with controllable reasoning that toggles on or off per request.
The memory footprint trips up most first-time deployers. Even though only 12B parameters activate per token, all 120B parameters must be loaded into VRAM, because any expert can be selected at any step. Nemotron 3 Super 120B carries the full memory burden of a 120B dense model, not a 12B one.
See how Nemotron 3 Super stacks up against other top models in our guide to the best open-source LLMs.
Nemotron 3 Super 120B GPU Requirements by Precision
VRAM requirements for Nemotron 3 Super 120B vary significantly by precision. The table below aligns the standard weights math with NVIDIA's own configuration counts, so the numbers match what NVIDIA actually ships.
| Precision | Approx. VRAM (weights + overhead) | Minimum GPU Configuration | Best For |
|---|---|---|---|
| BF16 (full) | ~270GB | 4x H100 or A100 80GB (320GB total) | Maximum accuracy |
| FP8 | ~150GB | 2x H100 or A100 80GB (32K context cap) | Balanced production inference |
| NVFP4 (native) | ~80GB | 1x 96GB Blackwell (RTX PRO 6000) | Efficient single-GPU inference |
| Q4_K_M (GGUF) | ~90GB | 1x 96GB Blackwell (RTX PRO 6000) | Local and community setups |
| Figures include model weights plus KV cache and activation headroom at moderate context length. VRAM scales with context; add margin for long-context workloads. | |||
BF16 needs 4x 80GB cards; FP8 fits on 2x. NVFP4 is a Blackwell-native format and runs on a single 96GB RTX PRO 6000. On an H100 (Hopper), use a 4-bit GGUF quant instead. 4-bit quantization roughly quarters VRAM relative to BF16, making single-GPU inference possible for a model this size.
What NVIDIA Officially Recommends
NVIDIA's published NIM deployment requirements are the most reliable hardware target for Nemotron 3 Super 120B. For a single-node self-hosted deployment, NVIDIA documents a 3-GPU minimum: 3x H100, 3x B200, or 3x RTX PRO 6000. Cloud instances come in 1, 2, 4, or 8 GPU configurations, so the practical rental target for full BF16 precision is 4x 80GB cards, which also provides comfortable headroom.
A lighter FP8 tensor-parallel profile runs on 2 GPUs, with context capped at 32,768 tokens. Supported microarchitectures are Ampere (A100), Hopper (H100 80GB), and Blackwell (B200, RTX PRO 6000). The model was pre-trained using NVFP4 quantization.
Full compatibility details are on the NVIDIA model card, the NVIDIA deployment docs, and the Hugging Face model card.
How to Deploy Nemotron 3 Super 120B With vLLM
vLLM with tensor parallelism is the standard way to serve Nemotron 3 Super 120B across multiple GPUs. Set --tensor-parallel-size to your GPU count, match --quantization to your chosen precision, and cap context when running the 2-GPU FP8 profile. The command below launches FP8 on 2 GPUs.
vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 \
--tensor-parallel-size 2 \
--quantization fp8 \
--gpu-memory-utilization 0.92 \
--max-model-len 32768 \
--trust-remote-code \
--port 8000
Two flags are non-obvious. --trust-remote-code is required because NVIDIA ships custom model code in the checkpoint. --max-model-len 32768 keeps the 2-GPU FP8 profile within memory. For BF16 on 4 GPUs, set --tensor-parallel-size 4 and drop --quantization, then confirm the exact checkpoint name on the model card before pulling weights.
Nemotron 3 Super 120B supports per-request reasoning control via an adjustable reasoning budget. Disable thinking mode for maximum throughput on simple tasks, or allocate a larger budget for complex multi-step problems.
Renting vs Buying GPUs for Nemotron 3 Super 120B
Renting multi-GPU cloud instances is cheaper than buying hardware for almost every Nemotron 3 Super 120B workload. The RTX PRO 6000 Blackwell (96GB) is the most accessible card for a self-host build, since H100 and A100 supply is largely allocation-gated. The RTX PRO 6000 launched at ~$8,565 in early 2025, but amid 2026 memory shortages has been listed as high as $16,000.
A100 and H100 instances on Thunder Compute are both on NVIDIA's supported hardware list for this model, and carry no upfront cost. The table below compares a self-host build against cloud rental for both the FP8 and BF16 configurations.
| Route | Configuration | Total VRAM | Upfront Cost | Ongoing Cost |
|---|---|---|---|---|
| Buy 2x RTX PRO 6000 | FP8 (32K context) | 192GB | ~$32K in cards, plus server build | Power, cooling, depreciation |
| Buy 3x RTX PRO 6000 | BF16 full precision | 288GB | ~$48K in cards, plus server build | Power, cooling, depreciation |
| Rent 2x H100 (Thunder Compute) | FP8 (32K context) | 160GB | $0 | ~$6.40/hr, billed per minute |
| Rent 4x A100 80GB (Thunder Compute) | BF16 full precision | 320GB | $0 | $5.96, billed per minute |
A 3x RTX PRO 6000 BF16 rig costs around $60K, while 4x A100 on Thunder Compute costs $5.96/hr. At that rate, you would need over 10,000 GPU-hours of continuous runtime, over a year nonstop, to reach the hardware's purchase price before accounting for power, cooling, and depreciation.
Running Nemotron 3 Super 120B on Thunder Compute
Thunder Compute offers A100 80GB and H100 instances, billed per minute. A 2x or 4x setup maps directly onto the FP8 and BF16 configurations in the requirements table, with no minimum commitment.
VS Code, Cursor, and Devin Desktop extensions let you develop against a live GPU instance directly from your editor. Persistent storage keeps model weights and environment state between sessions. To change GPU count, snapshot the current instance and launch a new one with a different configuration.
Last Thoughts on Running Nemotron 3 Super 120B
Match precision to hardware: 4x 80GB GPUs for full BF16, 2x for FP8, or a single 96GB Blackwell card at NVFP4. NVIDIA's published requirements give you a reliable baseline, and vLLM with tensor parallelism gets inference running in a few commands. With RTX PRO 6000 prices inflated by the 2026 memory shortage, renting A100 or H100 instances is the faster and cheaper path for nearly every workload.
How much VRAM does Nemotron 3 Super 120B need?
Nemotron 3 Super 120B needs ~270GB VRAM at BF16, ~150GB at FP8, and ~80GB at NVFP4. All 120B parameters must be loaded into VRAM regardless of precision, because it is an MoE model and any expert can activate at any step. Quantization is the main lever for reducing the required GPU count.
Can I run Nemotron 3 Super 120B on a consumer GPU like the RTX 4090 or 5090?
No. Even a 4-bit quantized version needs 70-90GB of VRAM, exceeding the 24GB on an RTX 4090 or the 32GB on an RTX 5090. Nemotron 3 Super 120B requires data-center GPUs such as the A100 or H100, owned or rented.
What is the cheapest way to run Nemotron 3 Super 120B?
Renting multi-GPU A100 or H100 instances by the hour is cheaper than buying hardware for most teams. Four A100 80GB GPUs on Thunder Compute cost $5.96 combined, with no upfront spend. Buying only amortizes at near-constant utilization over multiple years.
How many GPUs does Nemotron 3 Super 120B need?
NVIDIA documents a 3-GPU minimum: 3x H100, 3x B200, or 3x RTX PRO 6000. On cloud, plan for 4x 80GB GPUs for BF16 to fit standard instance sizes and add headroom. FP8 runs on 2 GPUs with context capped at 32,768 tokens; NVFP4 fits on a single 96GB Blackwell card.
Does Nemotron 3 Super 120B support reasoning control?
Yes. Nemotron 3 Super 120B supports per-request reasoning control via an adjustable reasoning budget. Disable thinking mode for maximum throughput on simple tasks, or allocate a larger budget for complex multi-step problems.