VRAM and memory bandwidth determine the right GPU for LLM work, not raw compute. The wrong choice means models don't load, inference stalls, or fine-tuning costs spiral. The right choice is not the most powerful GPU; it is the one that matches your workload, model size, and budget.
This guide covers local inference, QLoRA fine-tuning, and full training. It includes VRAM math, model-to-hardware mappings, cloud pricing, and inference runtime comparisons.
Two Specs That Decide Everything: VRAM and Bandwidth
VRAM determines whether your model loads at all. Memory bandwidth determines how fast tokens are generated. TFLOPS, CUDA core count, and clock speed are secondary for LLM workloads.
How to Calculate VRAM Requirements for Any Model
The baseline rule is 2 bytes per parameter at FP16. Q8 cuts this roughly in half; Q4 cuts it to about one quarter. A 7B model at FP16 needs approximately 14GB; at Q4 it fits under 6GB. A 70B model at FP16 needs roughly 140GB; at Q4 it fits in around 40GB.
The working formula is: VRAM = (parameters in billions × bytes per weight) × 1.2 overhead.
| Format | Bytes per Weight |
|---|---|
| FP32 (full precision) | 4.0 |
| BF16 / FP16 (half precision) | 2.0 |
| Q8 / INT8 | 1.0 |
| Q6_K | 0.75 |
| Q5_K_M | 0.625 |
| Q4_K_M | 0.5 |
| Q3_K_M | 0.375 |
| Q2_K | 0.25 |
VRAM Requirements by Model Size and Quantization
| Model size | FP16 | Q8 | Q4 | Fits on (Q4) |
|---|---|---|---|---|
| 7B | ~14GB | ~8GB | ~5GB | RTX 3060 12GB, RTX 4070 16GB |
| 13B | ~26GB | ~14GB | ~8GB | RTX 3060 12GB (tight), RTX 4070 Ti 16GB |
| 32B | ~65GB | ~35GB | ~20GB | RTX 5090 32GB, RTX 4090 24GB (tight) |
| 70B | ~140GB | ~75GB | ~40GB | A100 80GB, H100 80GB, dual RTX 5090 |
| 109B MoE (e.g. Llama 4 Scout) | ~220GB+ | ~110GB | ~55GB | A100 80GB (with Unsloth QLoRA), H100 80GB |
| 671B MoE (e.g. DeepSeek V3) | ~1.3TB | ~670GB | ~380GB | 8x H100/H200 nodes |
Why KV Cache Increases VRAM Requirements
The table above covers model weights only. During inference, the model stores a KV cache that grows with context length: roughly 0.5-1GB per 1K tokens for a 7B model.
A 70B model with a 128K context window can consume 30-60GB of additional VRAM at higher batch sizes. Estimate an extra 10-20% on top of weight VRAM for typical context lengths, more for long prompts or concurrent users.
Inference vs. Fine-Tuning vs. Training: Different Jobs, Different GPUs
Inference, QLoRA fine-tuning, and full training have different hardware requirements. Choosing the wrong tier is the most common and expensive GPU mistake.
Inference Is Bandwidth-Bound
LLM inference spends most of its time moving model weights from GPU memory to compute units, not doing arithmetic. Memory bandwidth is therefore the primary driver of throughput.
A GPU with 80GB of HBM at 2 TB/s generates tokens faster than one with the same VRAM and lower bandwidth, even at equal TFLOPS. This is why data center GPUs with HBM outperform consumer cards on large-model inference despite often having lower raw compute.
QLoRA Fine-Tuning: 3-4x the VRAM of Inference
Full fine-tuning stores gradients, optimizer states (Adam requires two moment tensors totaling ~2x the model weight memory), and activations on top of the model weights. Full fine-tuning a 7B model at FP16 requires roughly 60-80GB.
QLoRA reduces this dramatically by freezing the base model in 4-bit and training only small adapter layers. A 7B model under QLoRA fits in 10-12GB; Llama 4 Scout (109B total, 17B active) fits in 70-75GB on an A100 80GB with Unsloth.
For the complete setup and training script, see the Thunder Compute guide to fine-tuning Llama 4 Scout.
Full Training: Why Most Teams Should Use Cloud
Full pre-training at scale requires NVLink-connected GPU clusters where gradient synchronization completes in microseconds. This rules out consumer hardware for anything above 8B parameters.
For teams doing full training or RL runs above 70B, cloud is not just the cheaper option; it is the only practical one.
Estimating LLM Inference Speed by GPU
LLM inference is memory-bandwidth-bound during single-stream decode. The theoretical tokens/second ceiling is:
Tokens/sec ceiling = memory bandwidth (GB/s) ÷ model weight size (GB)
Real-world throughput typically runs at 60-80% of this ceiling.
| GPU | Memory Bandwidth | 7B FP16 ceiling | 13B FP16 ceiling | 70B FP16 ceiling |
|---|---|---|---|---|
| RTX A6000 | 768 GB/s | ~55 tok/s | ~30 tok/s | Does not fit |
| RTX 4090 | 1,008 GB/s | ~72 tok/s | ~39 tok/s | Does not fit |
| RTX 5090 | 1,792 GB/s | ~128 tok/s | ~69 tok/s | Does not fit |
| A100 80GB SXM | 2,039 GB/s | ~145 tok/s | ~78 tok/s | ~15 tok/s |
| H100 SXM | 3,350 GB/s | ~239 tok/s | ~129 tok/s | ~24 tok/s |
| H200 SXM | 4,800 GB/s | ~343 tok/s | ~185 tok/s | ~34 tok/s |
| Ceiling figures are theoretical maximums derived from published NVIDIA bandwidth specs.. | ||||
For training throughput, the workload is compute-bound rather than bandwidth-bound, making this formula inapplicable. MLPerf Training publishes standardised cross-vendor benchmarks for training workloads.
Which LLM Engine Runtime Should You Use?
The inference runtime matters as much as the GPU for real-world throughput. Text Generation Inference (TGI) moved to maintenance mode, announced in December 2025 with the GitHub repository archived read-only on March 21, 2026, redirecting new users to vLLM, SGLang, llama.cpp, and MLX.
- llama.cpp and Ollama are the right starting points for single-user local inference. Ollama wraps llama.cpp in a local API server with automatic VRAM management and a simple one-command model download. Both run on CPU if no GPU is available.
- vLLM is the standard choice for multi-user serving and production inference endpoints. It supports continuous batching, tensor parallelism across multiple GPUs, and an OpenAI-compatible API.
- SGLang delivers lower latency than vLLM on constrained-decoding workloads (structured JSON output, function calling). For general serving without structured output requirements, the difference is small enough that runtime familiarity often drives the choice.
- TensorRT-LLM delivers the highest sustained throughput on NVIDIA hardware for production workloads but requires more configuration, best suited to teams with dedicated ML infrastructure engineers.
Best GPU for Local LLM Inference
The right GPU for local inference depends entirely on the model size you are targeting.
Consumer Tier: RTX 5090 (32GB) and Mid-Range Options
The RTX 5090 is the current top consumer GPU for local LLM inference, with 32GB of GDDR7 and 1,792 GB/s bandwidth. It handles 32B models at Q4 on a single card reaching an acceptable 28 tokens/second.
The RTX 4090 (24GB) remains capable for 7B-32B models but is discontinued with inflated street prices. For budget builds, a used RTX 3090 (24GB) handles everything up to 32B at Q4 for around $400-600.
For mid-range local inference, the RTX 4070 Ti Super (16GB) is the best VRAM-per-dollar option below the 5090, handling 7B at full precision and 13B-14B at Q4.
Workstation Tier: RTX PRO 6000 Blackwell (96GB)
The RTX PRO 6000 Blackwell carries 96GB of GDDR7 ECC memory, enough to run 70B models at FP8 on a single card with 20GB+ of KV cache headroom. Workstation builds with this card start around $20,000-22,000 configured, so the economics favor cloud for most individual developers.
When Local Hardware Stops Making Sense
Local hardware becomes impractical for 70B+ models at FP16, multi-user serving, and QLoRA fine-tuning on models above 13B. The RTX 5090 cannot run 70B models at full precision on a single card. Without NVLink (the RTX 3090 was the last consumer card to support it), multi-GPU setups transfer data over PCIe at roughly half the throughput of NVLink-connected data center pairs.
Best Cloud GPU Instances for LLM Work
Cloud GPU removes the hardware purchase, maintenance, and obsolescence costs, and lets you change the hardware per project.
A100 80GB: The Value Pick for Inference and QLoRA Fine-Tuning
The A100 80GB is the most cost-effective cloud GPU for the majority of LLM workloads. Its 80GB of HBM2e handles 70B models at Q4, covers Llama 4 Scout fine-tuning with Unsloth, and serves multi-user inference for models up to 34B at FP16. The A100 costs 40-60% less/hr than an H100 and has a mature ecosystem for vLLM, Ollama, and Unsloth. Per the NVIDIA A100 datasheet, the SXM variant delivers 2,039 GB/s of HBM2e bandwidth; the PCIe variant delivers 1,935 GB/s.
On Thunder Compute, an A100 80GB runs at $1.09/hr. A 2-hour Llama 4 Scout fine-tuning run costs $2.18. A full day of 70B model inference costs around $26.
H100 80GB: When to Step Up for Larger Models
The H100 80GB delivers roughly 2-4x higher inference throughput than the A100 on bandwidth-heavy workloads. The SXM variant delivers 3.35 TB/s memory bandwidth versus the PCIe A100's 1,935 GB/s. It also adds FP8 support and features NVLink for multi-GPU distributed training at near-linear scale.
The H100 earns its cost when you are serving concurrent users at high throughput, running models at FP8 precision, or doing multi-GPU training where interconnect speed matters.
On Thunder Compute, an H100 PCIe runs at $2.19/hr.
H200 141GB: For 70B at FP16 and Long-Context Serving
The H200 carries 141GB of HBM3e at 4.8 TB/s bandwidth. It is the practical single-card option for serving 70B models at full FP16 precision and for long-context inference where KV cache dominates VRAM usage. For teams already on H100 infrastructure, it fits the same SXM5 server socket.
The H200 makes sense when the H100's 80GB ceiling is the constraint, for large-model single-card inference or for contexts where KV cache growth pushes past what the H100 can hold.
L40: Best Cost-per-Token for 7B-13B Inference
The L40 is the most cost-effective cloud GPU for serving 7B-13B models at scale. Its 48GB of GDDR6 and Ada Lovelace deliver lower cost-per-token than the A100 on these model sizes at batch sizes of 8+.
The L40 does not support NVLink, making it less suitable for multi-GPU distributed training. For inference on 7B-13B models and for computer vision pipelines that do not require large VRAM, it is the value pick.
Cloud GPU Pricing Comparison
| Provider | A100 80GB ($/hr) | H100 80GB ($/hr) | Notes |
|---|---|---|---|
| Thunder Compute | $1.09 | $2.19 | Per-minute billing, VS Code integration, persistent storage |
| Runpod | $1.39 | $2.89 | Secure Cloud rates |
| Lambda Labs | $2.79 | $3.29 | On-demand; H100 SXM available at higher rate |
| CoreWeave | $2.50 | $2.70 | Per-GPU rate normalized from 8-GPU node pricing; enterprise focus |
| AWS | $3.40 | $7.86 | On-demand p4/p5 instances; hyperscaler pricing |
See current GPU availability and pricing on Thunder Compute →
GPU Recommendations by Model (2026 Reference Table)
| Model | Active params | Min VRAM (Q4) | Recommended GPU | Cloud cost (Thunder) |
|---|---|---|---|---|
| Llama 3.1 8B | 8B (dense) | ~5GB | RTX A6000 | $0.35/hr |
| Llama 4 Scout | 17B (109B total MoE) | ~55GB | A100 80GB | $1.09/hr |
| Llama 3.1 70B | 70B (dense) | ~40GB | A100 80GB | $1.09/hr |
| DeepSeek R1 32B distill | 32B (dense) | ~20GB | A100 80GB | $1.09/hr |
| Qwen3 32B | 32B (dense) | ~20GB | A100 80GB | $1.09/hr |
| Qwen3 235B MoE | ~22B active | ~140GB | 2x A100 80GB | $2.18/hr |
| DeepSeek R1 (671B MoE) | ~37B active | ~376GB (full model) | 8x H100/H200 | Multi-GPU node |
| Kimi K2 (1T MoE) | 32B active | ~8x H200 (FP8) | 8x H200 node | Multi-GPU node |
| VRAM estimates at Q4 quantization for inference. Fine-tuning requires more; see the QLoRA section. MoE runtime VRAM is based on active parameters. | ||||
For a full breakdown of today's leading open-source models, see the Thunder Compute guide to the best open-source LLMs.
Best GPU for LLM Fine-Tuning with QLoRA
QLoRA freezes the base model in 4-bit precision and trains small adapter layers in full precision, cutting VRAM requirements by 3-4x compared to full fine-tuning.
A100 80GB: Fine-Tune Llama 4 Scout
The A100 80GB is the best-value GPU for QLoRA fine-tuning. Combined with Unsloth, it handles Llama 4 Scout at approximately 70-75GB VRAM usage, fitting within the 80GB ceiling.
A typical 2-hour run on a custom dataset costs $2.18. The A100 also covers Llama 3.1 70B QLoRA, Qwen3 32B, and any 7B-34B dense model with headroom to spare. See the step-by-step fine-tuning guide for commands, VRAM breakdown, and cost estimates.
H100: When to Step Up for Larger Fine-Tuning Jobs
The H100 earns its higher rate when you need faster iteration on large datasets, are running full SFT on 8B-30B models, or are using RL methods like GRPO or DPO that benefit from higher compute throughput.
For standard QLoRA runs on models up to 70B, the A100 delivers the same output at lower cost per run.
For a deeper treatment of the supervised fine-tuning pipeline, including DPO and preference alignment, see the supervised fine-tuning guide.
Which GPU Performs Best for LLM Training at Scale?
Full pre-training and large-scale RLHF require NVLink-connected GPU clusters where gradient synchronization happens in microseconds. The H100 SXM, H200, and B200 are the standard options, each with higher HBM bandwidth and NVLink throughput than PCIe variants. The B200 is the current standard for training frontier-scale models.
For most teams, full training above 8B parameters is a cloud-only workload. Owning an H100 SXM node starts around $300,000-400,000 in hardware alone. Thunder Compute supports up to 8-GPU instances for larger training and inference configurations.
Should You Buy or Rent?
Cloud GPU is cheaper than buying hardware for most developers. An RTX 5090 costs $4,000-5,000 at current street prices. At $1.09/hr for an A100 80GB on Thunder Compute, that purchase price covers around 3,650-4,600 hours of cloud compute, equivalent to 8 hours of daily GPU use for 1-2 years.
When cloud GPU wins: bursty workloads, testing different GPU types without committing to hardware, and fine-tuning jobs that don't fit on a consumer card. Per-minute billing means a 3-hour fine-tuning session on an A100 costs $3.27; a stopped instance costs nothing.
When on-premise makes sense: teams running GPU workloads more than 8-12 hours/day continuously with the infrastructure to manage it. Sustained inference serving at scale is the clearest on-premise use case.
Last Thoughts on the Best GPU for LLM
VRAM and memory bandwidth drive the GPU decision for LLM work, not raw compute. Identify your model size and quantization target, match it to the minimum GPU tier that covers it, then choose cloud or local based on how many hours per day the hardware will actually run.
For most developers doing inference and QLoRA fine-tuning, an A100 80GB in the cloud covers the full range of practical workloads at a fraction of hardware costs. The VS Code and Cursor extensions on Thunder Compute connect directly from your editor without SSH configuration.
See current GPU availability on Thunder Compute at thundercompute.com/pricing.
FAQ
How Much VRAM Do I Need for LLM Inference?
Plan on 2 bytes per parameter at FP16: a 7B model needs ~14GB, a 70B model needs ~140GB. Q4 quantization cuts this to ~0.5 bytes per parameter. Add 10-20% for KV cache overhead.
Which Inference Engine Is Best for LLMs?
llama.cpp and Ollama for single-user local inference. vLLM or SGLang for multi-user serving; vLLM has broader model support, SGLang has lower latency on constrained-decoding workloads. TensorRT-LLM delivers the highest sustained NVIDIA throughput but requires more configuration. TGI moved to maintenance mode in December 2025.
How Fast Does an A100 80GB Run a 7B LLM?
The theoretical ceiling for the A100 80GB SXM is ~145 tok/s for a 7B model at FP16, derived from the A100 SXM's 2,039 GB/s bandwidth divided by the 14 GB model weight size. The PCIe A100 (1,935 GB/s) yields ~138 tok/s. Real-world vLLM throughput runs at 60-80% of that ceiling.
Is the RTX 4090 Good for LLM Inference?
Yes, for 7B-32B models at Q4. The RTX 4090 is discontinued with inflated street prices. The RTX 5090 (32GB GDDR7) is the current consumer replacement. For models above 32B, a cloud A100 80GB is more practical and often more cost-effective.
What Is the Difference Between A100 and H100 for LLM Work?
Both have 80GB of VRAM. The H100 SXM delivers 3.35 TB/s bandwidth and adds FP8 and NVLink for multi-GPU training. The A100 costs 40-60% less/hr and handles most inference and QLoRA fine-tuning without needing the H100's extra throughput.
Which GPU Do I Need to Run a 70B Model?
An A100 80GB handles a 70B model at Q4 (~40GB) on a single card. Consumer options require dual RTX 5090s or the RTX PRO 6000 Blackwell (96GB). A cloud A100 80GB on Thunder Compute starts at $1.09/hr.
What GPU Is Best for QLoRA Fine-Tuning?
The A100 80GB is the best-value option. It handles Llama 4 Scout (109B total, 17B active) with Unsloth at 70-75GB VRAM. A 2-hour fine-tuning run costs $2.18 on Thunder Compute.
Should I Buy a GPU or Rent Cloud GPU for LLM Work?
Cloud wins for most developers. An RTX 5090 costs $4,000-5,000; a cloud A100 80GB runs $1.09/hr on Thunder Compute. Unless you run workloads more than 8 hours per day continuously, cloud is cheaper and removes maintenance overhead.
Do I Need NVIDIA for LLMs, or Will AMD Work?
NVIDIA is significantly easier. CUDA covers every major inference and fine-tuning framework without extra configuration. AMD ROCm supports Ollama natively on RDNA3 and RDNA4, but NVIDIA remains the default for vLLM, Unsloth, and production workflows.
When Does Local Hardware Beat Cloud GPU for LLMs?
Local hardware makes sense only for teams running GPU workloads more than 8-12 hrs/day continuously. For bursty workloads, experimentation, or fine-tuning, cloud per-minute billing is cheaper and removes driver and maintenance overhead.