Choosing a GPU for AI work comes down to one question: will your model fit, and will it run fast enough to be practical? The answer depends on your workload, your model size, and whether you have data center hardware or a consumer card.
Most teams make the same two mistakes. They either underprovision VRAM and watch their training run crash halfway through. Or they overprovision, renting an H100 for experiments that would complete on an A100 for half the cost.
Key Takeaways
- Match VRAM to model size first. Inference in FP16 needs approximately 2 bytes per parameter. Fine-tuning with LoRA adds 50% headroom. Full training with Adam needs roughly 16 bytes per parameter.
- Memory bandwidth matters more than TFLOPS for most AI workloads. An A100 at 2,039 GB/s will outperform a card with higher raw compute but slower memory on large-model inference.
- Scaling usually means moving to 4 to 8 GPUs, not switching to a single larger card.
- Renting cloud GPUs is almost always cheaper than buying data center hardware for sporadic or experimental workloads.
- Test in the cloud before committing to hardware. Two hours on an A100 costs about $2.18 and tells you everything a spec sheet won't.
.png)
Why GPU Selection Determines Project Success
The wrong GPU wastes money in both directions:
- An underpowered card extends training jobs, driving up total compute cost even at a lower hourly rate.
- An overpowered card burns budget on unused capacity.
GPU choice also determines iteration speed. A fine-tune that takes 2 hours on an A100 might take 10 hours on an RTX A6000. Faster hardware produces faster model improvement, faster debugging cycles, and faster deployment.
VRAM limits are the most common project killer. When a model doesn't fit in GPU memory, training crashes, and no optimization trick will work.
Understand Your Workload First
GPU requirements differ dramatically between workload types. The same model can have a 10x difference in VRAM requirements depending on whether you are running inference, fine-tuning, or training from scratch.
GPU for LLM Training
Full training from scratch is rare outside large research teams. It is also extremely VRAM-intensive: a 7B parameter model requires approximately 112 GB with a standard Adam optimizer in mixed precision, making a multi-GPU setup mandatory. For most teams, fine-tuning a pre-trained model is the practical path.
LLM training prioritizes VRAM capacity first, memory bandwidth second, and raw compute third. Bandwidth-limited cards create bottlenecks that additional TFLOPS cannot resolve.
GPU for Fine-Tuning
Fine-tuning with LoRA or QLoRA cuts VRAM requirements dramatically compared to full training. A 7B model fine-tuned with LoRA fits on a 24 to 40 GB card. A 13B model fits on an A100 40GB. Even 30B-class models become feasible on a single 48 GB card with QLoRA at 4-bit quantization.
The practical starting point for most fine-tuning work is a single A100 80GB. It handles 7B to 30B models comfortably in LoRA without quantization and scales to multi-GPU via NVLink when needed.
Compare A100 and H100 for fine-tuning and multi-GPU training workloads.
GPU for AI Inference
Inference needs enough VRAM to load the model, then optimizes for throughput and latency. A 7B model requires approximately 14 GB in FP16. Once loaded, the GPU needs to generate tokens quickly and cost-efficiently.
For production inference serving concurrent users, memory bandwidth drives throughput more than compute. At batch size 1, memory bandwidth is the primary bottleneck. At batch sizes of 8 or above, FP8 compute throughput from the Transformer Engine on L40S and H100 class GPUs delivers substantially lower cost per token.
Research and Experimentation
Research workloads benefit from flexibility more than any specific hardware optimization. You may be training a small model one day and loading a large pre-trained transformer for inference the next. The ability to switch GPU types without losing your environment matters more than peak performance on any single task.
Thunder Compute's VS Code and Cursor extensions let you connect to any GPU instance directly from your editor. To switch GPU types, save an instance snapshot and launch a new instance from it.
The Three GPU Specs That Actually Matter
Most GPU spec sheets lead with CUDA core counts and clock speeds. For AI workloads, neither matters much. Three specs do.
VRAM Capacity for AI Models
VRAM is the hard limit. Your model weights, gradients, optimizer states, activations, and KV cache all compete for the same pool of GPU memory. Exceed it and training crashes. There is no workaround beyond CPU offloading, which can reduce throughput significantly depending on the operation. Size VRAM to your workload before looking at any other spec.
Memory Bandwidth in AI Training
Memory bandwidth is the spec most developers skip, and it is often the most important. Most AI workloads are memory-bound, not compute-bound: your GPU spends more time moving data between memory and compute cores than it does executing math. A card that can move data faster will complete memory-bound training steps proportionally faster, regardless of raw TFLOPS.
This is why an A100 with 2,039 GB/s bandwidth consistently outperforms an RTX 4090 with 1,008 GB/s bandwidth on large model inference: the A100's HBM2e keeps compute cores fed continuously while the 4090's GDDR6X creates memory bottlenecks at large model sizes.
Compute Precision: FP16, BF16, and FP8
Mixed precision training in FP16 or BF16 is standard practice on every modern GPU. FP8 halves the memory footprint of model weights compared to FP16, enabling higher batch sizes and lower cost per token for inference.
FP8 with automatic per-layer precision switching via NVIDIA's Transformer Engine is available on H100 and L40S class GPUs. A100 and older cards support FP16 and BF16 only. For large-scale inference on 30B or 70B models, the H100 at FP8 can deliver up to 4 times the token throughput of an A100 at FP16.
GPU VRAM Requirements by Model Size
The table below shows approximate VRAM requirements for common AI models at each workload stage.
| Model | Inference VRAM (FP16) | Fine-Tuning VRAM (LoRA/QLoRA)1 | Full Training VRAM2 |
|---|---|---|---|
| GPT-2 (1.5B) | 3 GB | ~6 GB | ~24 GB |
| Llama 3.1 7B / 8B | ~14 GB | ~24 GB (40 GB recommended) | ~112 GB (multi-GPU) |
| Llama 3.1 13B | ~26 GB | ~40 GB | ~208 GB (multi-GPU) |
| Qwen3 32B (FP16) | ~64 GB | ~80 GB+ (QLoRA: ~20 GB) | Multi-GPU required |
| Llama 3.1 70B (FP16) | ~140 GB | Multi-GPU (QLoRA: ~35 GB single card) | Multi-GPU required |
| 1Estimates assume 4-bit base model quantization.
2Estimates assume FP16 mixed precision with Adam optimizer: approximately 16 bytes per parameter for weights, gradients, and optimizer states, before activation memory. Actual usage varies with batch size and sequence length. |
|||
When fine-tuning pushes past single-GPU limits, the practical next step is 4-8 GPUs with NVLink, not switching to a larger single card.
GPU Options for AI Workloads
Different GPU architectures serve different roles. Choosing the right tier avoids both underpowering your workload and spending on capabilities you will never use.
RTX A6000 for Development and Small-Scale Inference
The RTX A6000 is the right GPU for inference workloads, development, and experimentation on models up to 13B parameters. Its 48 GB of GDDR6 handles 7B models comfortably in FP16 and makes QLoRA fine-tuning of 13B models practical on a single card.
At 768 GB/s memory bandwidth, the A6000 is slower than data center cards for large-model training. For production training on 30B or larger models, it will be the bottleneck. Use it for development and smaller workloads, then scale to A100 or H100 for production.
Compare the RTX A6000 and A100 for fine-tuning and inference workloads.
L40 and L40S for Mid-Scale Inference
The L40 is a strong mid-tier option for inference and computer vision pipelines. It carries 48 GB of GDDR6 ECC and Ada Lovelace fourth-generation Tensor Cores, giving it higher AI TOPS than the A6000 while handling a broader range of workloads.
The L40S is the AI-optimized variant: same die, same VRAM, but with the Transformer Engine enabled and higher clock speeds. The L40S doubles FP8 throughput to 733 TFLOPS dense and makes FP8 inference automatic in vLLM. For image generation, VDI, and low-batch FP16 inference, the L40's lower hourly rate makes it the better value.
Neither the L40 nor L40S supports NVLink. For multi-GPU training on large models, A100 or H100 SXM configurations are more efficient.
See the full L40 vs L40S comparison with FP8 benchmarks and cost-per-token data.
A100 80GB for LLM Fine-Tuning and Production Training
The A100 80GB is the practical workhorse for most AI training workloads. Its 80 GB of HBM2e and 2,039 GB/s memory bandwidth handle fine-tuning of models up to 30B parameters on a single card, and it scales efficiently to 4 to 8 GPUs via NVLink for larger runs. Third-generation Tensor Cores provide strong FP16 and BF16 throughput across every major training framework.
The A100 also supports Multi-Instance GPU (MIG) partitioning, which splits a single card into up to seven isolated instances for multi-tenant inference or shared development environments.
Explore full A100 specs, MIG configuration, and fine-tuning guidance.
H100 for Large Model Training and Fast Inference
The H100 is built for the largest and most demanding AI workloads. Its 3,350 GB/s memory bandwidth (SXM variant) is 64% higher than the A100 SXM, and its Transformer Engine with FP8 delivers 3 to 4 times the inference throughput for 70B models. Fourth-generation Tensor Cores and NVLink 4.0 at 900 GB/s make it the right card for multi-GPU distributed training at scale.
Despite a higher hourly rate, the H100 often delivers lower total job cost for training runs on 30B or larger models, because faster completion reduces total GPU hours consumed.
See the A100 vs H100 per-job cost breakdown with worked examples.
H200 and B200 for 70B+ Workloads and Next-Generation Training
The H200 carries 141 GB of HBM3e at 4.8 TB/s bandwidth, making it the practical choice for serving 70B models at FP16 on a single card and for long-context inference where KV cache dominates VRAM usage. It fits the same SXM5 server socket as the H100, making it a straightforward upgrade for teams on H100 infrastructure.
The B200 (Blackwell architecture) sits at the current top of the performance ladder, with 192 GB of HBM3e at 8 TB/s bandwidth. It suits frontier model training and very large inference deployments. B200 availability through cloud providers is improving in 2026, though it remains more constrained than H100 or H200.
| GPU | VRAM | Memory Bandwidth1 | FP16 Tensor TFLOPS | Used Price2 | Cloud Pricing | Best For |
|---|---|---|---|---|---|---|
| RTX 4090 | 24 GB GDDR6X | 1,008 GB/s | 82.6 (FP32)3 | $1,800–$2,500 | Buy only (consumer card) | Local dev, 7B inference |
| RTX A6000 | 48 GB GDDR6 | 768 GB/s | 309.7 | $3,500–$5,990 | $0.35–$2.12/hr | Inference, dev, small fine-tunes |
| L40 | 48 GB GDDR6 | 864 GB/s | 181 | — | $0.56–$1.25/hr | Mid-scale inference, CV pipelines |
| A100 80GB | 80 GB HBM2e | 2,039 GB/s | 312 | $4,800–$9,000 | $1.09–$3.43/hr | LLM fine-tuning, production training |
| H100 SXM 80GB | 80 GB HBM3 | 3,350 GB/s | 989 | $12,000–$22,000 | $1.38–$11.06/hr | Large model training, fast inference |
| H200 SXM 141GB | 141 GB HBM3e | 4,800 GB/s | 989 | — | $2.40–$10.60/hr | 70B+ inference, long-context serving |
| B200 | 192 GB HBM3e | 8,000 GB/s | 2,2504 | — | $3.50–$27.04/hr | Frontier model training, max throughput |
| Sources: NVIDIA A100 Datasheet; NVIDIA L40S Product Page; NVIDIA H100 Product Page.
1Bandwidth figures are for SXM variants where applicable. 2Used card prices are secondary-market estimates for single units. 3RTX 4090 figure shown is FP32 CUDA throughput. 4B200 FP16 Tensor performance with sparsity. |
||||||
The bandwidth gap between GDDR6-based GPUs and HBM-based data center cards explains why an A100 often outperforms cards with lower raw Tensor TFLOPS on large-model inference. AI training is mostly memory-bound, so faster data movement translates directly into faster iteration.
Should You Rent or Buy a GPU for AI?
Renting cloud GPUs is the right default for most AI teams. Data center GPUs like the A100 and H100 cost $10,000 to $30,000 or more to purchase, require data center power and cooling, and depreciate as new generations arrive.
Buying makes sense only for consumer cards running sustained, continuous inference. An RTX 4090 can serve as a local inference server for 7B to 13B models, and at equivalent cloud rental rates, a purchased unit may pay for itself over several months of continuous use.
The calculation depends heavily on current retail prices and your actual utilization rate. For training, fine-tuning, and anything requiring data center hardware, renting is cheaper.
The practical approach most teams settle on is hybrid: develop on lower-cost cloud instances, run training jobs on cloud GPUs scaled to the workload, and optionally deploy inference on owned consumer hardware.
Multi-GPU Scaling for AI Workloads
Moving from one GPU to multiple GPUs is the right path when a single card cannot fit your model, or when single-card training is too slow for your iteration timeline. Multi-GPU setups add complexity and do not always scale linearly.
Data parallelism distributes training batches across multiple GPUs, with each GPU holding a full copy of the model. It suits models that fit on a single card but need faster training on larger datasets. Model parallelism splits the model itself across cards and is required for models too large for any single GPU. Most teams start with data parallelism because it requires minimal changes to existing training code.
NVLink makes multi-GPU scaling significantly more efficient for A100 and H100 configurations. NVLink 3.0 on A100 delivers 600 GB/s bidirectional bandwidth; NVLink 4.0 on H100 delivers 900 GB/s. PCIe multi-GPU setups (L40, L40S) work for inference parallelism but become communication bottlenecks for large-scale training, where PCIe 4.0 x16 tops out at approximately 64 GB/s bidirectional.
Common GPU Selection Mistakes
Most GPU selection mistakes come from applying the wrong mental model to the problem.
Choosing based on gaming benchmarks or TFLOPS numbers. Gaming benchmarks measure a completely different workload than AI training. TFLOPS figures across architectures are not directly comparable because each architecture executes tensor operations differently. The only reliable comparison is real benchmarks on the framework and model you will actually use, or memory bandwidth figures, which predict memory-bound AI performance more reliably.
Underestimating VRAM requirements. Inference VRAM looks manageable, but fine-tuning and training add optimizer states, gradients, and activation memory. A 7B model needs roughly 14 GB for inference and approximately 112 GB for full training. Use the full training formula (around 16 bytes per parameter) when sizing hardware.
Ignoring memory bandwidth. A card with higher raw TFLOPS but lower bandwidth will be slower on most AI workloads. The A100 has far higher Tensor Core TFLOPS and significantly higher bandwidth than the RTX 4090, which is why it consistently outperforms it on large model inference despite being older.
Buying hardware before testing your workload. Hardware purchased based on spec sheets frequently does not match real-world performance for a specific model and training configuration. Rent a cloud instance for a few hours, run an actual workload to measure GPU utilization and memory usage.
GPU Performance Monitoring and Optimization
A GPU that looks right on paper can be underutilized due to CPU bottlenecks, data loading inefficiencies, or suboptimal batch sizes. Monitoring is as important as selection.
Track four metrics during training runs: GPU utilization (target above 80%), memory utilization (target 85 to 95% of available VRAM), training throughput in samples or tokens per second, and cost per epoch. Low GPU utilization alongside high memory utilization typically indicates bandwidth saturation. Low utilization with low memory usage points to a CPU or data pipeline bottleneck.
Gradient accumulation, mixed precision training, and gradient checkpointing are the standard tools for improving VRAM efficiency when you cannot scale hardware. Gradient checkpointing trades compute for memory by recomputing activations during the backward pass instead of storing them.
Running Your Workload on Thunder Compute
Thunder Compute offers on-demand GPU instances across four tiers: RTX A6000 from $0.35/hr, L40 from $0.79/hr, A100 80GB from $1.09/hr, and H100 PCIe from $2.19/hr. All instances include persistent storage and multiple vCPU/RAM configurations, and are billed by the minute with no minimum commitment.
The VS Code and Cursor extensions connect directly from your editor to a running instance, eliminating SSH configuration and container setup. To switch between GPU tiers, save an instance snapshot and launch a new instance from it.
Last Thoughts on Choosing the Right GPU for AI Workloads
Calculate your VRAM floor, identify your workload type, match it to the right hardware tier, and run a quick cloud test before committing. For the vast majority of AI teams, cloud GPUs are the right starting point: they eliminate upfront cost, provide top tier data center hardware, and let you scale for a single training job.
FAQ
How Do I Calculate How Much VRAM I Need for My AI Model?
Inference in FP16 needs approximately 2 bytes per parameter: a 7B model needs roughly 14 GB. LoRA fine-tuning adds about 50% headroom over that. Full training with Adam in mixed precision needs approximately 16 bytes per parameter, so a 7B model requires around 112 GB, making multi-GPU mandatory.
What Is the Difference Between GPU Requirements for Training vs Inference?
Training needs maximum VRAM for weights, gradients, and optimizer states, plus high bandwidth to keep compute fed during long jobs. Inference only needs enough VRAM to load the model once, then optimizes for latency and cost-per-token. For the same model, training can require 4 to 8 times more VRAM than inference.
Should I Rent or Buy a GPU for AI Work?
Rent when workloads are sporadic, when you need data center GPUs that cost $10K to $30K or more to own, or when you want to test before committing. Buy consumer cards only when running continuous inference and the purchase price pays off against cloud billing within a few months. For training and experimentation, renting is cheaper.
Why Does Memory Bandwidth Matter More Than TFLOPS for AI?
Most AI workloads are memory-bound: GPUs spend more time moving data than doing math. A card with faster bandwidth keeps compute cores fed and reduces idle cycles. This is why an A100 at 2,039 GB/s consistently outperforms lower-bandwidth cards on large model inference, even when those cards have higher consumer benchmark scores.
What GPU Do I Need to Fine-Tune a 7B Model?
An RTX A6000 (48 GB) or L40 (48 GB) handles 7B LoRA fine-tuning comfortably. QLoRA at 4-bit quantization brings requirements down to 16 to 24 GB, making an RTX 4090 workable. For full fine-tuning without LoRA or for 13B models, an A100 80GB is the reliable single-card option.
When Should I Move From a Single GPU to Multiple GPUs?
Move to multi-GPU when your model does not fit in a single card's VRAM at the required precision, or when single-card training is too slow. Start with data parallelism across 2 to 4 GPUs before attempting model parallelism, which requires more significant code changes. A100 and H100 with NVLink scale multi-GPU workloads more efficiently than PCIe-only cards like the L40 or L40S.
What Is the Difference Between the L40 and L40S for AI Workloads?
The L40S has the Transformer Engine enabled, doubling FP8 throughput to 733 TFLOPS dense and making FP8 inference automatic via vLLM's --dtype fp8 flag. The L40 requires manual FP8 configuration per layer. At batch size 8 or above the L40S wins on cost per token. For image generation, VDI, and low-batch FP16 inference, the L40's lower rate wins.
Is the H100 Worth It Over the A100 for Most Teams?
For training over 30B parameters or serving 70B models at production throughput, yes. The H100's FP8 support and 64% higher memory bandwidth (SXM variant) typically reduce total job cost by 30 to 40% despite the higher hourly rate. For fine-tuning under 30B or development work, the A100 is usually cheaper per job.
What Is MIG Partitioning and How Does It Help with GPU Sharing?
MIG (Multi-Instance GPU) splits a single A100 or H100 into up to seven isolated instances, each with dedicated VRAM, cache, and compute cores. It lets multiple developers or inference workloads share one GPU without interfering with each other, reducing per-seat costs in team environments.
What Is Gradient Checkpointing and When Should I Use It?
Gradient checkpointing recomputes activations during the backward pass instead of storing them, trading extra compute for lower VRAM usage. Use it when hitting VRAM limits during training and you cannot scale hardware. It typically increases training time by 20 to 30% while cutting activation memory significantly.