The right GPU for an AI workload comes down to two questions: will your model fit, and will it run fast enough to be practical? VRAM sets the floor; memory bandwidth determines throughput. Getting either wrong costs money.
Most teams make the same two mistakes: underprovision VRAM and watch training crash halfway through, or overprovision and rent an H100 for experiments that would complete on an A100 for half the cost.
Key Takeaways
- Match VRAM to model size first. Inference in FP16 needs approximately 2 bytes per parameter. Fine-tuning with LoRA adds 50% headroom. Full training with Adam needs roughly 16 bytes per parameter.
- Memory bandwidth matters more than TFLOPS for most AI workloads. An A100 at 2,039 GB/s will outperform a card with higher raw compute but slower memory on large-model inference.
- Renting cloud GPUs is always cheaper than buying data center hardware for sporadic or experimental workloads.
GPU Selection Framework
.png)
Understand Your Workload First
GPU requirements differ dramatically between workload types. The same model can have a 10x difference in VRAM requirements depending on whether you are running inference, fine-tuning, or training from scratch.
Quick Reference: Workload Sizing Summary
| Workload | What determines your requirement | How to calculate it |
|---|---|---|
| Inference (FP16) | Model weight size + KV cache | 2 bytes per parameter + 10-20% overhead |
| LoRA / QLoRA fine-tuning | Quantized base model + adapter gradients | Inference VRAM + 50% headroom |
| Full training (Adam, FP16) | Weights + gradients + optimizer states + activations | ~16 bytes per parameter (before activation memory) |
| Multi-GPU scaling | Model fit, interconnect type, parallelism strategy | Start with data parallelism; use NVLink cards for model parallelism |
GPU for LLM Training
Full training from scratch is extremely VRAM-intensive. A 7B parameter model requires approximately 112 GB with a standard Adam optimizer in mixed precision, making a multi-GPU setup mandatory. For most teams, fine-tuning a pre-trained model is the practical path.
LLM training prioritizes VRAM capacity first, memory bandwidth second, and raw compute third. Bandwidth-limited cards create bottlenecks that additional TFLOPS cannot resolve.
GPU for Fine-Tuning
Fine-tuning with LoRA or QLoRA cuts VRAM requirements dramatically compared to full training. A 7B model fine-tuned with LoRA fits on a 24-40 GB card. A 13B model fits on an A100 40GB. Even 30B-class models become feasible on a single 48 GB card with QLoRA at 4-bit quantization.
The practical starting point for most fine-tuning work is a single A100 80GB. It handles 7B-30B models comfortably in LoRA without quantization and scales to multi-GPU via NVLink when needed.
Compare A100 and H100 for fine-tuning and multi-GPU training workloads.
GPU for AI Inference
Inference needs enough VRAM to load the model, then optimizes for throughput and latency. A 7B model requires approximately 14 GB in FP16. Memory bandwidth drives throughput at batch size 1; at batch sizes of 8 or above, FP8 compute throughput from the Transformer Engine on L40S and H100 class GPUs delivers substantially lower cost-per-token.
Research and Experimentation
Research workloads benefit from flexibility more than any specific hardware optimization. The ability to switch GPU types without losing your environment matters more than peak performance on any single task.
Thunder Compute's VS Code and Cursor extensions connect directly from your editor to a running instance. To switch GPU types, save an instance snapshot and launch a new instance with a different configuration. Your installed packages, custom scripts, and files carry over.
Factors to Consider When Selecting GPUs for AI
CUDA core counts and clock speeds dominate GPU spec sheets. For AI workloads, neither matters much. The factors that actually drive your decision are:
- VRAM
- Memory bandwidth
- Compute power
VRAM Capacity for AI Models
VRAM is the hard limit. Model weights, gradients, optimizer states, activations, and KV cache all compete for the same pool of GPU memory. Exceed it and training crashes with no workaround beyond CPU offloading, which can reduce throughput significantly. Size VRAM to your workload before looking at any other spec.
Memory Bandwidth in AI Training
Memory bandwidth is the spec most developers skip, and it is often the most important. Research from Brookhaven National Laboratory confirms that most AI workloads are memory-bound, not compute-bound: GPUs spend more time moving data between memory and compute cores than executing math.
An A100 SXM with 2,039 GB/s bandwidth (PCIe: 1,935 GB/s) consistently outperforms an RTX 4090 with 1,008 GB/s on large model inference. The A100's HBM2e keeps compute cores fed continuously while the 4090's GDDR6X creates memory bottlenecks at large model sizes.
Compute Precision: FP16, BF16, and FP8
Mixed precision training in FP16 or BF16 is standard practice on every modern GPU. FP8 halves the memory footprint of model weights compared to FP16, enabling higher batch sizes and lower cost-per-token for inference.
FP8 with automatic per-layer precision switching via NVIDIA's Transformer Engine is available on H100 and L40S class GPUs. A100 and older cards support FP16 and BF16 only. For large-scale inference on 30B or 70B models, the H100 at FP8 can deliver up to 4x the token throughput of an A100 at FP16.
Ecosystem and Software Compatibility
NVIDIA CUDA remains the path of least resistance for every major AI framework. PyTorch, vLLM, Unsloth, and TensorRT-LLM all treat NVIDIA hardware as the default target. AMD ROCm supports Ollama natively on RDNA3 and RDNA4, but framework coverage is narrower and some optimization libraries have CUDA-only implementations.
Power, Cooling, and Infrastructure Constraints
High-end GPUs pull 300-450W under sustained load. Before purchasing a GPU for on-premise use, confirm that your power supply, circuit capacity, and cooling can handle the sustained load. An H100 SXM requires liquid cooling and purpose-built infrastructure. For most teams, these constraints make cloud the practical choice for data-center-class hardware.
How to Calculate VRAM Requirements for Your Workload
The VRAM math differs by workload stage. The formulas below give the starting estimate; add 10-20% overhead for KV cache (inference) or activation memory (training).
Inference (FP16): VRAM = parameters (billions) × 2 bytes × 1.2
LoRA / QLoRA fine-tuning: VRAM = inference VRAM × 1.5 (base model in 4-bit + adapter gradients)
Full training with Adam (FP16): VRAM = parameters (billions) × 16 bytes (weights + gradients + optimizer states, before activation memory)
| Model | Inference VRAM (FP16) | Fine-Tuning VRAM (LoRA/QLoRA)1 | Full Training VRAM2 |
|---|---|---|---|
| GPT-2 (1.5B) | 3 GB | ~6 GB | ~24 GB |
| Llama 3.1 7B / 8B | ~14 GB | ~24 GB (40 GB recommended) | ~112 GB (multi-GPU) |
| Llama 3.1 13B | ~26 GB | ~40 GB | ~208 GB (multi-GPU) |
| Qwen3 32B (FP16) | ~64 GB | ~80 GB+ (QLoRA: ~20 GB) | Multi-GPU required |
| Llama 3.1 70B (FP16) | ~140 GB | Multi-GPU (QLoRA: ~35 GB single card) | Multi-GPU required |
| 1Estimates assume 4-bit base model quantization.
2Estimates assume FP16 mixed precision with Adam optimizer: approximately 16 bytes per parameter for weights, gradients, and optimizer states, before activation memory. Actual usage varies with batch size and sequence length. |
|||
For specific GPU hardware recommendations by workload, see the Best GPU for AI guide. For LLM-specific GPU recommendations, see the Best GPU for LLM guide.
GPU Options for AI Workloads
Different GPU architectures serve different roles. Choosing the right tier avoids both underpowering your workload and spending on capabilities you will never use.
RTX A6000 for Development and Small-Scale Inference
The RTX A6000 is the right GPU for inference workloads, development, and experimentation on models up to 13B parameters. Its 48 GB of GDDR6 handles 7B models comfortably in FP16 and makes QLoRA fine-tuning of 13B models practical on a single card.
At 768 GB/s memory bandwidth, the A6000 is slower than data center cards for large-model training. For production training on 30B or larger models, it will be the bottleneck. Use it for development and smaller workloads, then scale to A100 or H100 for production.
Compare the RTX A6000 and A100 for fine-tuning and inference workloads.
L40 and L40S for Mid-Scale Inference
The L40 is a strong mid-tier option for inference and computer vision pipelines. It carries 48 GB of GDDR6 ECC and Ada Lovelace 4th-gen Tensor Cores, giving it higher AI TOPS than the A6000 while handling a broader range of workloads.
The L40S is the AI-optimized variant: same die, same VRAM, but with the Transformer Engine enabled and higher clock speeds. The L40S doubles FP8 throughput to 733 TFLOPS dense and makes FP8 inference automatic in vLLM. For image generation, VDI, and low-batch FP16 inference, the L40's lower hourly rate makes it the better value.
Neither the L40 nor L40S supports NVLink. For multi-GPU training on large models, A100 or H100 SXM configurations are more efficient.
Compare the L40 vs L40S or the L40 and A100 to choose the right mid-tier GPU for your inference or fine-tuning workload.
A100 80GB for LLM Fine-Tuning and Production Training
The A100 80GB is the practical workhorse for most AI training workloads. Its 80 GB of HBM2e and 2,039 GB/s memory bandwidth handle fine-tuning of models up to 30B parameters on a single card, and it scales efficiently to 4-8 GPUs via NVLink for larger runs. 3rd-gen Tensor Cores provide strong FP16 and BF16 throughput across every major training framework.
The A100 also supports MIG (Multi-Instance GPU) partitioning, which splits a single card into up to seven isolated instances for multi-tenant inference or shared development environments.
Explore full A100 specs, MIG configuration, and fine-tuning guidance.
H100 for Large Model Training and Fast Inference
The H100 is built for the largest and most demanding AI workloads. Its 3,350 GB/s memory bandwidth (SXM variant) is 64% higher than the A100 SXM, and its Transformer Engine with FP8 delivers 3-4x the inference throughput for 70B models. 4th-gen Tensor Cores and NVLink 4.0 at 900 GB/s make it the right card for multi-GPU distributed training at scale.
Despite a higher hourly rate, the H100 often delivers lower total job cost for training runs on 30B or larger models, because faster completion reduces total GPU hours consumed.
See the A100 vs H100 per-job cost breakdown with worked examples.
H200 and B200 for 70B+ Workloads and Next-Generation Training
The H200 carries 141 GB of HBM3e at 4.8 TB/s bandwidth, making it the practical choice for serving 70B models at FP16 on a single card and for long-context inference where KV cache dominates VRAM usage. It fits the same SXM5 server socket as the H100, making it a straightforward upgrade for teams on H100 infrastructure.
The B200 (Blackwell architecture) sits at the current top of the performance ladder, with 192 GB of HBM3e at 8 TB/s bandwidth. It suits frontier model training and very large inference deployments. B200 availability through cloud providers is improving in 2026, though it remains more constrained than H100 or H200.
| GPU | VRAM | Memory Bandwidth1 | FP16 Tensor TFLOPS | Used Price2 | Cloud Pricing | Best For |
|---|---|---|---|---|---|---|
| RTX 4090 | 24 GB GDDR6X | 1,008 GB/s | 82.6 (FP32)3 | $1,800-$2,500 | Buy only (consumer card) | Local dev, 7B inference |
| RTX A6000 | 48 GB GDDR6 | 768 GB/s | 309.7 | $3,500-$5,990 | $0.35-$2.12/hr | Inference, dev, small fine-tunes |
| L40 | 48 GB GDDR6 | 864 GB/s | 181 | — | $0.53-$1.25/hr | Mid-scale inference, CV pipelines |
| A100 80GB | 80 GB HBM2e | 2,039 GB/s | 312 | $4,800-$9,000 | $1.09-$3.43/hr | LLM fine-tuning, production training |
| H100 SXM 80GB | 80 GB HBM3 | 3,350 GB/s | 989 | $12,000-$22,000 | $1.38-$11.06/hr | Large model training, fast inference |
| H200 SXM 141GB | 141 GB HBM3e | 4,800 GB/s | 989 | — | $3.44-$10.60/hr | 70B+ inference, long-context serving |
| B200 | 192 GB HBM3e | 8,000 GB/s | 2,2504 | — | $3.50-$27.04/hr | Frontier model training, max throughput |
| Prices as of August 2026. Single-GPU on-demand rates where available. CoreWeave rate normalized from 8-GPU node pricing. | ||||||
Multi-GPU Scaling for AI Workloads
Moving from one GPU to multiple GPUs is the right path when a single card cannot fit your model, or when single-card training is too slow for your iteration timeline.
Data parallelism distributes training batches across multiple GPUs, with each GPU holding a full copy of the model. It suits models that fit on a single card but need faster training on larger datasets. Model parallelism splits the model itself across cards and is required for models too large for any single GPU. Most teams start with data parallelism because it requires minimal changes to existing training code.
NVLink makes multi-GPU scaling significantly more efficient for A100 and H100 configurations. Per the A100 datasheet, NVLink 3.0 on A100 delivers 600 GB/s bidirectional bandwidth; NVLink 4.0 on H100 delivers 900 GB/s. PCIe multi-GPU setups work for inference parallelism but become communication bottlenecks for large-scale training, where PCIe 4.0 x16 tops out at approximately 64 GB/s bidirectional.
Common GPU Selection Mistakes
Choosing based on gaming benchmarks or TFLOPS numbers. Gaming benchmarks measure a completely different workload than AI training. TFLOPS figures across architectures are not directly comparable because each architecture executes tensor operations differently. Memory bandwidth figures predict memory-bound AI performance more reliably than peak TFLOPS.
Underestimating VRAM requirements. Inference VRAM looks manageable, but fine-tuning and training add optimizer states, gradients, and activation memory. A 7B model needs roughly 14 GB for inference and approximately 112 GB for full training. Use the full training formula when sizing hardware for training workloads.
Ignoring memory bandwidth. A card with higher raw TFLOPS but lower bandwidth will be slower on most AI workloads. The A100 has far higher bandwidth than the RTX 4090, which is why it consistently outperforms it on large model inference despite being an older design.
Buying hardware before testing your workload. Hardware purchased based on spec sheets frequently does not match real-world performance for a specific model and training configuration. Rent a cloud instance for a few hours, run an actual workload to measure GPU utilization and memory usage, then commit.
How to Benchmark a GPU for AI Workloads
The only reliable pre-purchase test is running your actual workload. Rent a cloud instance for 1-2 hours and run your training or inference job for a fixed number of steps.
Track GPU utilization (target above 80%), memory utilization (target 85-95%), training throughput in samples or tokens/second, and cost per epoch. Low GPU utilization alongside high memory utilization indicates bandwidth saturation. Low utilization with low memory usage points to a CPU or data pipeline bottleneck.
For standardized cross-vendor reference points on training throughput, MLPerf publishes independently audited benchmarks across GPU hardware.
How to Track and Compare Training Experiments Across GPU Hardware
Consistent experiment logging and environment preservation are the two requirements for meaningful hardware comparisons.
Experiment tracking tools like Weights and Biases and MLflow let you log GPU utilization, memory usage, training throughput, and cost per epoch across runs with different hardware configurations. Tag each run with the GPU type to make comparisons straightforward.
On Thunder Compute, save an instance snapshot and launch a new instance from it with a different GPU configuration. The snapshot carries over your installed packages, datasets, and training scripts, so the only variable between runs is the hardware.
GPU Performance Monitoring and Optimization
Track four metrics during training runs: GPU utilization (target above 80%), memory utilization (target 85-95% of available VRAM), training throughput in samples or tokens/second, and cost per epoch. Low GPU utilization alongside high memory utilization typically indicates bandwidth saturation. Low utilization with low memory usage points to a CPU or data pipeline bottleneck.
Gradient accumulation, mixed precision training, and gradient checkpointing are the standard tools for improving VRAM efficiency when you cannot scale hardware. Gradient checkpointing recomputes activations during the backward pass instead of storing them, typically increasing training time by 20-30% while cutting activation memory significantly.
Running Your Workload on Thunder Compute
Thunder Compute offers on-demand GPU instances across four tiers: RTX A6000 from $0.35/hr, L40 from $0.79/hr, A100 80GB from $1.09/hr, and H100 PCIe from $2.19/hr. All instances include persistent storage and multiple vCPU/RAM configurations, and are billed by the minute with no minimum commitment.
The VS Code and Cursor extensions connect directly from your editor to a running instance, eliminating SSH configuration and container setup. To switch between GPU tiers, save an instance snapshot and launch a new instance from it.
Last Thoughts on Choosing the Right GPU for AI Workloads
Calculate your VRAM floor, identify your workload type, match it to the right hardware tier, and run a quick cloud test before committing. For the vast majority of AI teams, cloud GPUs are the right starting point: they eliminate upfront cost, provide top-tier data center hardware, and let you scale for a single training job.
FAQ
How Do I Calculate How Much VRAM I Need for My AI Model?
Inference in FP16 needs ~2 bytes per parameter: a 7B model needs ~14 GB. LoRA fine-tuning adds ~50% headroom. Full training with Adam in mixed precision needs ~16 bytes per parameter, so a 7B model requires ~112 GB, making multi-GPU mandatory.
What Is the Difference Between GPU Requirements for Training vs Inference?
Training needs maximum VRAM for weights, gradients, and optimizer states, plus high bandwidth to keep compute fed during long jobs. Inference needs enough VRAM to load the model once, then optimizes for latency and cost-per-token. For the same model, training can require 4-8x more VRAM than inference.
Should I Rent or Buy a GPU for AI Work?
Rent when workloads are sporadic, when you need data center GPUs that cost $10K-$30K+ to own, or when you want to test before committing. Buy consumer cards only when running continuous inference and the purchase price pays off against cloud billing within a few months. For training and experimentation, renting is cheaper.
Why Does Memory Bandwidth Matter More Than TFLOPS for AI?
Most AI workloads are memory-bound: GPUs spend more time moving data than doing math. An A100 SXM at 2,039 GB/s consistently outperforms lower-bandwidth cards on large model inference, even when those cards have higher consumer benchmark scores.
When Should I Move From a Single GPU to Multiple GPUs?
Move to multi-GPU when your model does not fit in a single card's VRAM at the required precision, or when single-card training is too slow. Start with data parallelism across 2-4 GPUs before attempting model parallelism. A100 and H100 with NVLink scale multi-GPU workloads more efficiently than PCIe-only cards like the L40 or L40S.
How Do I Track and Compare Training Experiments Across GPU Hardware?
Use Weights and Biases or MLflow to log GPU utilization, memory usage, training throughput, and cost per epoch on each run. On Thunder Compute, save an instance snapshot and launch a new instance with a different GPU configuration from it, letting you run the same training job on an A6000, A100, and H100 without rebuilding your environment each time.
How Do I Benchmark a GPU Before Buying for AI Training?
Run your actual training or inference job for a short fixed duration on a rented instance before committing to hardware. Track GPU utilization, memory utilization, and throughput in samples or tokens/second. MLPerf Training publishes standardized cross-vendor benchmarks if you want third-party reference points. Low GPU utilization in practice usually indicates a data pipeline or batch size problem, not a GPU problem.
What Is Gradient Checkpointing and When Should I Use It?
Gradient checkpointing recomputes activations during the backward pass instead of storing them, trading extra compute for lower VRAM usage. Use it when hitting VRAM limits during training and you cannot scale hardware. It typically increases training time by 20-30% while cutting activation memory significantly.
What Is the Difference Between FP16, BF16, and FP8 for AI Training?
FP16 and BF16 are standard mixed-precision formats on every modern GPU. BF16 handles a wider numerical range than FP16, making it more stable for large model training. FP8 halves the memory footprint of FP16, enabling higher batch sizes and lower cost-per-token for inference. FP8 with automatic precision switching via NVIDIA's Transformer Engine is available on H100 and L40S class GPUs only.
How Do I Monitor GPU Performance During Training?
Track GPU utilization (target 80%+), memory utilization (target 85-95% of available VRAM), training throughput in samples or tokens/second, and cost per epoch. Low GPU utilization alongside high memory utilization typically indicates bandwidth saturation. Low utilization with low memory usage points to a CPU or data pipeline bottleneck.