Choosing the GPU for your LLM work is the biggest hardware decision in any project. The wrong choice means models don't load, inference stalls, or fine-tuning costs spiral. The right choice is not the most powerful GPU; it is the one that matches your workload, model size, and budget.
This guide gives recommendations for major use cases: local inference, QLoRA fine-tuning, and full training. It covers the VRAM math behind every decision, maps leading open-source models to the right hardware, and compares cloud GPU pricing so you can make an informed decision.
Two Specs That Decide Everything: VRAM and Bandwidth
VRAM determines whether your model loads at all. Memory bandwidth determines how fast tokens are generated. For LLM workloads, every other GPU spec, TFLOPS, CUDA core count, clock speed, is secondary to these two numbers.
How to Calculate VRAM Requirements for Any Model
The baseline rule is 2GB of VRAM per billion parameters at FP16 precision. Q8 cuts this roughly in half; Q4 cuts it to about one quarter. A 7B model at FP16 needs approximately 14GB; the same model at Q4 fits under 6GB. A 70B model at FP16 needs roughly 140GB; at Q4 it fits in around 40GB.
The working formula is: VRAM = (parameters in billions × bytes per weight) × 1.2 overhead.
| Format | Bytes per Weight |
|---|---|
| FP32 (full precision) | 4.0 |
| BF16 / FP16 (half precision) | 2.0 |
| Q8 / INT8 | 1.0 |
| Q6_K | 0.75 |
| Q5_K_M | 0.625 |
| Q4_K_M | 0.5 |
| Q3_K_M | 0.375 |
| Q2_K | 0.25 |
VRAM Requirements by Model Size and Quantization
| Model size | FP16 | Q8 | Q4 | Fits on (Q4) |
|---|---|---|---|---|
| 7B | ~14GB | ~8GB | ~5GB | RTX 3060 12GB, RTX 4070 16GB |
| 13B | ~26GB | ~14GB | ~8GB | RTX 3060 12GB (tight), RTX 4070 Ti 16GB |
| 32B | ~65GB | ~35GB | ~20GB | RTX 5090 32GB, RTX 4090 24GB (tight) |
| 70B | ~140GB | ~75GB | ~40GB | A100 80GB, H100 80GB, dual RTX 5090 |
| 109B MoE (e.g. Llama 4 Scout) | ~220GB+ | ~110GB | ~55GB | A100 80GB (with Unsloth QLoRA), H100 80GB |
| 671B MoE (e.g. DeepSeek V3) | ~1.3TB | ~670GB | ~380GB | 8x H100/H200 nodes |
Why KV Cache Increases VRAM Requirements
The table above covers model weights only. During inference, the model stores a KV cache that grows with context length: roughly 0.5-1GB per 1K tokens for a 7B model.
A 70B model with a 128K context window can consume 30-60GB of additional VRAM at higher batch sizes. Estimate an extra 10-20% on top of weight for typical context lengths, more for long prompts or concurrent users.
Inference vs. Fine-Tuning vs. Training: Different Jobs, Different GPUs
Picking the wrong GPU tier for your workload is the most common and expensive GPU mistake. Inference, QLoRA fine-tuning, and full training each have different hardware requirements.
Inference Is Bandwidth-Bound
LLM inference spends most of its time moving model weights from GPU memory to compute units, not doing arithmetic. Memory bandwidth is therefore the primary driver of tokens-per-second throughput.
A GPU with 80GB of HBM at 2 TB/s generates tokens faster than one with the same VRAM and lower bandwidth, even at equal TFLOPS. This is why data center GPUs with HBM outperform consumer cards on large-model inference despite often having lower raw compute.
QLoRA Fine-Tuning: 3-4x the VRAM of Inference
Full fine-tuning stores gradients, optimizer states (Adam uses 2x the model weight memory), and activations on top of the model weights. Full fine-tuning a 7B model at FP16 requires roughly 60-80GB, far more than inference.
QLoRA reduces this dramatically by freezing the base model in 4-bit and training only small adapter layers. A 7B model under QLoRA fits in 10-12GB; Llama 4 Scout (109B total, 17B active) fits in 70-75GB on an A100 80GB with Unsloth.
For the complete setup and training script, see the Thunder Compute guide to fine-tuning Llama 4 Scout.
Full Training: Why Most Teams Should Use Cloud
Full pre-training at scale requires NVLink-connected GPU clusters where gradient synchronization completes in microseconds. This rules out consumer hardware for anything above 8B parameters.
For teams doing full training or RL runs above 70B, cloud is not just the cheaper option; it is the only practical one.
Best GPU for Local LLM Inference
Local inference means model weights and compute live on your machine. The right GPU depends entirely on the model size you are targeting.
Consumer Tier: RTX 5090 (32GB) and Mid-Range Options
The RTX 5090 is the current top consumer GPU for local LLM inference, with 32GB of GDDR7 and 1,792 GB/s bandwidth. It handles 32B models at Q4 on a single card and reaches around 185 tokens per second on 8B models.
The RTX 4090 (24GB) remains capable for 7B-32B models but is discontinued with inflated street prices. For budget builds, a used RTX 3090 (24GB) handles everything up to 32B at Q4 for around $400-600.
For mid-range local inference, the RTX 4070 Ti Super (16GB) is the best VRAM-per-dollar option below the 5090, handling 7B at full precision and 13B-14B at Q4.
Professional Tier: RTX PRO 6000 Blackwell (96GB) for Best GPU for Local LLM
The RTX PRO 6000 Blackwell is the first professional workstation GPU with 96GB of GDDR7 ECC memory, enough to run 70B models at FP8 on a single card with meaningful KV cache headroom.
It reaches around 829 tokens per second on 32B FP16 models via vLLM. Workstation builds with this card start around $20,000-22,000 configured, so the economics favor cloud for most individual developers.
When Local Hardware Stops Making Sense
Local hardware becomes impractical for 70B+ models at FP16, multi-user serving, and QLoRA fine-tuning on models above 13B. The RTX 5090 cannot run 70B models at full precision on a single card, and without NVLink (removed after the RTX 3090), multi-GPU setups transfer data over PCIe at roughly half the throughput of NVLink-connected data center pairs.
Beyond these limits, cloud GPU is the faster, simpler, and often cheaper path.
Best Cloud GPU for LLM Work
Cloud GPU removes the hardware purchase, maintenance, and obsolescence costs, and lets you change the hardware for each project.
A100 80GB: The Value Pick for Inference and QLoRA Fine-Tuning
The A100 80GB is the most cost-effective cloud GPU for the majority of LLM workloads. Its 80GB of HBM2e handles 70B models at Q4, covers Llama 4 Scout fine-tuning with Unsloth, and serves multi-user inference for models up to 34B at FP16. The A100 costs 40-60% less per hour than an H100 and has a mature software ecosystem for vLLM, Ollama, Unsloth, and TGI.
On Thunder Compute, an A100 80GB runs at $1.09 per GPU-hour. A 2-hour Llama 4 Scout fine-tuning run costs $2.18. A full day of 70B model inference costs around $26.
H100 80GB: When to Step Up for Larger Models
The H100 80GB delivers roughly 2-4x higher inference throughput than the A100 on bandwidth-heavy workloads, with 3.35 TB/s memory bandwidth on the SXM variant versus the A100's 2 TB/s. It also adds FP8 support and NVLink for multi-GPU distributed training at near-linear scale.
The H100 earns its cost when you are serving concurrent users at high throughput, running models at FP8 precision, or doing multi-GPU training where interconnect speed matters.
On Thunder Compute, an H100 PCIe runs at $2.19 per GPU-hour.
Cloud GPU Pricing Comparison: Which GPU to Rent for LLM Work
| Provider | A100 80GB ($/hr) | H100 80GB ($/hr) | Notes |
|---|---|---|---|
| Thunder Compute | $1.09 | $2.19 | Per-minute billing, VS Code integration, persistent storage |
| RunPod | $1.19 | $1.99 | Marketplace; rates vary by pod |
| Lambda Labs | $2.79 | $3.29 | On-demand; H100 SXM available at higher rate |
| CoreWeave | $2.50 | $2.70 | Normalized from 8-GPU node pricing; enterprise focus |
| AWS | $3.40 | $7.86 | On-demand p4/p5 instances; hyperscaler pricing |
| Prices as of June 2026 from provider pricing pages. Single-GPU on-demand rates in US regions. Multi-GPU node prices normalized to per-GPU equivalent. | |||
See current GPU availability and pricing on Thunder Compute →
GPU Recommendations by Model (2026 Reference Table)
The table below maps leading open-source LLMs to the minimum GPU required and the recommended cloud option.
| Model | Active params | Min VRAM (Q4) | Recommended GPU | Cloud cost (Thunder) |
|---|---|---|---|---|
| Llama 4 Scout | 17B (109B total MoE) | ~55GB | A100 80GB | $1.09/hr |
| Llama 3.1 70B | 70B (dense) | ~40GB | A100 80GB | $1.09/hr |
| Llama 3.1 8B | 8B (dense) | ~5GB | RTX 4070 Ti / A100 | $1.09/hr |
| DeepSeek R1 (671B MoE) | ~37B active | ~376GB (full model) | 8x H100/H200 | Multi-GPU node |
| DeepSeek R1 32B distill | 32B (dense) | ~20GB | A100 | $1.09/hr |
| Kimi K2 (1T MoE) | 32B active | ~8x H200 (FP8) | 8x H200 node | Multi-GPU node |
| Qwen3 32B | 32B (dense) | ~20GB | A100 | $1.09/hr |
| Qwen3 235B MoE | ~22B active | ~140GB | 2x A100 80GB | $2.18/hr |
| VRAM estimates at Q4 quantization for inference. Fine-tuning requires more; see the QLoRA section. MoE runtime VRAM is based on active parameters. | ||||
For a full breakdown of today's leading open-source models and benchmarks, see the Thunder Compute guide to the best open-source LLMs.
Best GPU for LLM Fine-Tuning with QLoRA
QLoRA is the standard technique for fine-tuning large models without the VRAM cost of full training. It freezes the base model in 4-bit precision and trains small adapter layers in full precision.
A100 80GB: Fine-Tune Llama 4 Scout
The A100 80GB is the best-value GPU for QLoRA fine-tuning. Combined with Unsloth, it handles Llama 4 Scout at approximately 70-75GB VRAM usage, fitting within the 80GB ceiling.
A typical 2-hour run on a custom dataset costs $2.18. The A100 also covers Llama 3.1 70B QLoRA, Qwen3 32B, and any 7B-34B dense model with headroom to spare. See the step-by-step fine-tuning guide for commands, VRAM breakdown, and cost estimates.
H100: When to Step Up for Larger Fine-Tuning Jobs
The H100 earns its higher rate when you need faster iteration on large datasets, are running full SFT on 8B-30B models, or are using RL methods like GRPO or DPO that benefit from higher compute throughput.
For standard QLoRA runs on models up to 70B, the A100 delivers the same output at lower cost per run.
For a deeper treatment of the supervised fine-tuning pipeline, including DPO and preference alignment, see the Thunder Compute supervised fine-tuning guide.
Which GPU Performs Best for LLM Training at Scale?
Full pre-training and large-scale RLHF require NVLink-connected GPU clusters where gradient synchronization happens in microseconds. The H100 SXM, H200, and B200 are the standard options, each with higher HBM bandwidth and NVLink throughput than PCIe variants. The B200 delivers roughly 4-5x the inference throughput of an H100 and is the current standard for training frontier-scale models.
For most teams, full training above 8B parameters is a cloud-only workload. Owning an H100 SXM node starts around $300,000-400,000 in hardware alone; renting by the hour removes that barrier entirely. Thunder Compute supports up to 8-GPU instances for larger training and inference configurations.
Should You Buy or Rent? The Break-Even Math for Best Budget GPU for Local LLM Inference
Cloud GPU is cheaper than buying hardware for most developers. An RTX 5090 costs $4,000-5,000 at current street prices, excluding the workstation, power supply, and cooling. At $1.09/hr for an A100 80GB on Thunder Compute, that purchase price covers around 3650-4600 hours of cloud compute; equivalent to 8 hours of daily GPU use for 1-2 years.
When Cloud GPU Wins
Cloud is the right call for bursty workloads, for developers who want to test different GPU types without committing to hardware, and for any fine-tuning job that doesn't fit on a consumer card.
Per-minute billing means you pay only for active compute time: a 3-hour fine-tuning session on an A100 costs $3.27; a stopped instance costs nothing.
When On-Premise Hardware Makes Sense
Buying hardware makes economic sense only for teams running GPU workloads more than 8-12 hours per day continuously and who have the infrastructure to manage it. Even then, the operational cost of power, cooling, driver management, and eventual replacement often exceeds the apparent savings.
Sustained inference serving at scale is the clearest on-premise use case; development, experimentation, and fine-tuning almost always favor cloud.
Last Thoughts on the Best GPU for LLM
VRAM and memory bandwidth drive the GPU decision for LLM work, not raw compute. Identify your model size and quantization target, match it to the minimum GPU tier that covers it, then choose cloud or local based on how many hours per day the hardware will actually run.
For most developers doing inference and QLoRA fine-tuning, an A100 80GB in the cloud covers the full range of practical workloads at a fraction of hardware costs.
Thunder Compute offers A100 80GB instances from $1.09/hr and H100 PCIe from $2.19/hr, with per-minute billing, persistent storage, and VS Code integration built in.
How much VRAM do I need for LLM inference?
Plan on 2GB per billion parameters at FP16. Q4 quantization reduces this to 0.5GB per billion: a 7B model needs ~5GB at Q4, a 70B model needs ~40GB. Add 10-20% for KV cache overhead.
Is the RTX 4090 good for LLM inference?
Yes, the RTX 4090 is good for 7B-32B models at Q4, but it is discontinued and street prices are inflated. The RTX 5090 (32GB GDDR7) is the current consumer replacement. For models above 32B, a cloud A100 80GB is more practical and often more cost-effective.
What is the difference between A100 and H100 for LLM work?
Both have 80GB of VRAM. The H100 delivers 2-4x higher bandwidth and adds FP8 and NVLink for multi-GPU training. The A100 costs 40-60% less per hour and handles most inference and QLoRA fine-tuning without needing the H100's extra throughput.
Which GPU do I need to run a 70B model?
A GPU with at least 40GB of VRAM handles a 70B model at Q4. The A100 80GB covers this on a single card. Consumer options require dual RTX 5090s or a RTX PRO 6000 Blackwell (96GB). A cloud A100 80GB on Thunder Compute starts at $0.78/hr.
What GPU is best for QLoRA fine-tuning?
The A100 80GB is the best-value option. It handles Llama 4 Scout (109B total, 17B active) with Unsloth at 70-75GB VRAM, completing a 2-hour fine-tuning run for under $2 on Thunder Compute.
Should I buy a GPU or rent cloud GPU for LLM work?
Cloud wins for most developers. An RTX 5090 costs $4,000-5,000 at street prices; a cloud A100 80GB runs $0.78/hr on Thunder Compute. Unless you run workloads more than 8 hours per day, every day, cloud is cheaper and removes maintenance overhead.
Do I need NVIDIA for LLMs, or will AMD work?
NVIDIA is significantly easier. CUDA covers every major inference and fine-tuning framework without extra configuration. AMD ROCm now supports Ollama natively on RDNA3 and RDNA4, but NVIDIA remains the default for vLLM, Unsloth, and production workflows.
What is the best budget GPU for local LLM inference?
A used RTX 3090 (24GB VRAM, ~$400-600) offers the best VRAM per dollar under $600 and handles 7B-32B models at Q4. For new hardware, the RTX 5070 Ti (16GB) is the best mid-range entry point in 2026.
When does local hardware beat cloud GPU for LLMs?
Local hardware makes sense only for teams running GPU workloads more than 8-12 hrs/day continuously. For bursty workloads, experimentation, or fine-tuning, cloud per-minute billing is cheaper and removes driver and maintenance overhead.