The NVIDIA L40 is a 48GB Ada Lovelace data center GPU released in October 2022, built for AI inference, image generation, and fine-tuning. It shares the AD102 silicon family with the L40S but uses a different die revision and clock configuration, making the two GPUs look similar on paper while performing differently in practice.
NVIDIA L40 at a Glance
| Specification | Value |
|---|---|
| Architecture | Ada Lovelace (AD102-895) |
| CUDA Cores | 18,176 |
| Tensor Cores | 568 (4th Gen) |
| RT Cores | 142 (3rd Gen) |
| GPU Memory | 48GB GDDR6 ECC |
| Memory Bandwidth | 864GB/s |
| FP32 Performance | 90.5 TFLOPS |
| FP16 Tensor Performance | 181 TFLOPS (362 TFLOPS with sparsity) |
| FP8 Tensor Performance | 362 TFLOPS (724 TFLOPS with sparsity) |
| L2 Cache | 96 MB |
| Base / Boost Clock | 735 MHz / 2,490 MHz |
| TDP | 300 W |
| Form Factor | PCIe Gen4 x16, dual-slot, full-height full-length |
| NVLink | No |
| MIG Support | No |
| Launch Date | October 2022 |
| Source: NVIDIA L40 Datasheet (January 2023). | |
Rent an L40 on Thunder Compute, $0.79/hr with per-minute billing and no commitments.
Architecture: Ada Lovelace and the AD102 Die
The L40 is built on NVIDIA's Ada Lovelace architecture, manufactured on TSMC's 4N process using the AD102-895 die. The same die family powers the L40S (AD102-896) and the RTX 4090.
CUDA Cores, Tensor Cores, and RT Cores
The L40 provides 18,176 CUDA cores across 142 streaming multiprocessors. Its 568 fourth-generation Tensor Cores support FP8, BF16, FP16, and TF32 precision, covering every major mixed-precision training and inference workflow. The 142 third-generation RT Cores handle hardware ray tracing, though this is not relevant for AI compute workloads.
FP8 and the Transformer Engine
The L40 supports FP8 at the hardware level, but its Transformer Engine is not enabled by default. The Transformer Engine dynamically switches between FP8 and BF16 at the layer level during inference and training. On the L40S, it is enabled out of the box; on the L40, FP8 is accessible but without that dynamic mixed-precision management.
This means the L40 delivers 362 TFLOPS at FP8 versus the L40S's 733 TFLOPS, despite identical silicon. The gap is in how FP8 is managed at inference time, not in hardware capability. For a full breakdown, see the L40 vs L40S comparison.
Memory Subsystem: 48GB GDDR6 and 864GB/s Bandwidth
The L40 uses 48GB of GDDR6 ECC memory across a 384-bit interface, delivering 864GB/s of bandwidth. This matches the L40S exactly, since both revisions share the same memory subsystem. The A100 80GB reaches 2,039GB/s via HBM2e, which is why it leads in bandwidth-bound training despite having fewer CUDA cores.
Performance: What You Get from 90.5 TFLOPS FP32
FP32, FP16, and FP8 TFLOPS
The L40 delivers 90.5 TFLOPS at FP32, 181 TFLOPS at FP16 Tensor (362 with sparsity), and 362 TFLOPS at FP8 (724 with sparsity). Inference workloads rarely run at FP32; the figures that matter for LLM serving are FP16 and FP8 Tensor performance. The L40S doubles both of those to 362 TFLOPS at FP16 and 733 TFLOPS at FP8.
LLM Inference Throughput
At batch size 1 in FP16, the L40 generates approximately 75-100 tokens/second on a 7B model and around 15-25 tokens/second on a 70B model at INT4. At batch size 8, throughput on 7B models scales to roughly 250-350 tokens/second in FP16. Results vary by quantization method, context length, and framework version.
Image Generation
The L40 handles Stable Diffusion XL within its 48GB with significant headroom. ComfyUI workflows with multiple ControlNet passes or video generation pipelines run on a single L40 without memory spilling, which is not always possible on 24GB cards.
Fine-Tuning
For LoRA and QLoRA fine-tuning of 7B to 13B models, the L40's 48GB removes most memory bottlenecks at typical batch sizes. Full fine-tuning of models above 13B parameters in FP16 will exhaust the available memory; QLoRA with 4-bit quantization is the standard workaround for larger models.
What Models Fit in 48GB?
The table below shows approximate VRAM requirements for common models. Figures cover model weights only; KV cache, activations, and framework overhead add to these totals at runtime.
| Model | Precision | Approximate VRAM | |
|---|---|---|---|
| Llama 3 8B | FP16 | ~16GB | |
| Llama 3 8B | INT4 (QLoRA) | ~5GB | |
| DeepSeek R1 32B | Q4 | ~18GB | |
| DeepSeek R1 70B | Q4 | ~33GB | |
| Llama 3 70B | INT4 | ~35GB | |
| Flux.1 Dev | FP16 | ~24GB | |
| Stable Diffusion XL | FP16 | ~7GB | |
| Wan 2.2 T2V-14B | FP16 | ~28GB | |
| VRAM estimates reflect model weights only. KV cache, activations, and framework overhead add additional requirements at runtime, particularly at larger batch sizes and context lengths. | |||
Use Cases
AI Inference (7B to 70B Models)
The L40 is well-suited for serving 7B to 13B models at FP16 and larger models at INT4. Its 48GB of VRAM holds Llama 3 70B at 4-bit quantization alongside a reasonable KV cache, making it a practical single-GPU inference server for most production API workloads. At batch sizes below 16, memory bandwidth is not yet the binding bottleneck, so the L40 delivers competitive tokens-per-dollar versus higher-bandwidth GPUs.
Image and Video Generation
Pipelines with multiple ControlNet passes, IP-Adapter layers, or high-resolution upscaling benefit from the L40's memory headroom. Flux.1 Dev and SDXL with full pipelines loaded stay within 48GB without offloading, reducing latency compared to the same pipeline on a 24GB card. Video generation models such as Wan and CogVideoX also fit within the L40's memory budget at standard resolutions.
Fine-Tuning and Small-Scale Training
LoRA and QLoRA fine-tuning of 7B to 13B models is a natural fit. The 48GB capacity removes most memory bottlenecks at typical training batch sizes, and FP8 support reduces memory pressure during forward passes. Full fine-tuning of larger models requires quantization or a multi-GPU setup; at that point the A100 SXM with NVLink is the more appropriate choice.
Limitations
No NVLink: What This Means for Multi-GPU Scaling
The L40 uses PCIe Gen4, providing up to 64GB/s of bidirectional bandwidth. NVLink on the A100 SXM provides 600GB/s per GPU; the H100 SXM scales this to 900GB/s per GPU with NVLink 4.0. For tensor parallelism across a multi-GPU training run, the PCIe bottleneck becomes significant. The L40 works well in single-GPU deployments and horizontally scaled inference clusters, but is not suited for tightly coupled distributed training.
No MIG Partitioning
The L40 does not support Multi-Instance GPU (MIG) partitioning. The full 48GB VRAM is allocated to a single workload per instance. For multi-tenant serving of many small models with hardware isolation, the A100 with MIG is the right tool.
NVIDIA L40 Cloud Pricing
Marketplace platforms offer lower starting rates but variable availability. Dedicated GPU cloud providers offer more predictable access with fixed pricing.
| Provider | L40 Price (per GPU/hr) | Notes |
|---|---|---|
| Thunder Compute | $0.79 | Per-minute billing, no commitments |
| RunPod | $0.99 | Marketplace, availability varies |
| Vast.ai | From $0.56 | Marketplace, host-dependent reliability |
| CoreWeave | $1.25 | 8-GPU node minimum |
| Pricing reflects public on-demand rates as of July 2026. Marketplace pricing fluctuates based on live availability. See the full L40 and L40S pricing guide for a complete provider breakdown. | ||
Thunder Compute bills per minute with no minimum session length, so short workloads, spot tests, and fine-tuning runs are only billed for actual usage.
Last Thoughts on the NVIDIA L40
The L40 is a practical choice for teams that need 48GB of VRAM for inference, image generation, or fine-tuning without paying for training-tier hardware. For higher inference throughput, the L40S is worth the premium. For distributed training with NVLink, the A100 SXM is the right tool. For most other workloads, the L40 at $0.79/hr on Thunder Compute delivers strong performance per dollar.
FAQ
How much VRAM does the NVIDIA L40 have?
The NVIDIA L40 has 48GB of GDDR6 ECC memory with 864GB/s of memory bandwidth. This is enough to run Llama 3 70B at INT4 quantization, Flux.1 Dev at FP16, and most image generation pipelines without CPU offloading.
Does the NVIDIA L40 support FP8?
Yes. The L40 reaches 362 TFLOPS at FP8 (724 TFLOPS with sparsity) via its fourth-generation Tensor Cores. Unlike the L40S, the Transformer Engine is not enabled by default, so it does not dynamically switch between FP8 and BF16 at the layer level during inference.
Does the NVIDIA L40 support NVLink or MIG?
No. The L40 supports neither NVLink nor MIG partitioning. Multi-GPU communication runs over PCIe Gen4. For workloads needing tight GPU-to-GPU interconnect or tenant isolation via MIG, the A100 SXM is the appropriate alternative.
Is the NVIDIA L40 good for LLM training?
The L40 is well-suited for LoRA and QLoRA fine-tuning of 7B to 13B models on a single GPU. For large-scale distributed training above 13B parameters in full precision, or workloads that require NVLink, the A100 SXM is the better choice.
What is the L40's TDP?
The NVIDIA L40 has a 300W TDP, lower than the L40S (350W) and the A100 SXM (400W). This makes it more power-efficient in rack-dense deployments where thermal headroom is a constraint.
How does the L40's FP16 performance compare to the A100?
The L40 delivers 181 TFLOPS at FP16 Tensor versus the A100 80GB's 312 TFLOPS. The A100's bigger advantage is memory bandwidth: 2,039GB/s via HBM2e versus the L40's 864GB/s via GDDR6. In single-GPU inference at moderate batch sizes, the L40's Ada Lovelace architecture narrows the gap.
What models can I run on the NVIDIA L40?
The L40's 48GB fits Llama 3 8B at FP16 (16GB), Llama 3 70B at INT4 (35GB), Mistral NeMo 12B at FP16 (24GB), Flux.1 Dev at FP16 (24GB), and SDXL at FP16 (~7GB). Models requiring more than 48GB at full precision need quantization or multi-GPU setup.