The NVIDIA L40 is a 48GB Ada Lovelace data center GPU built for AI inference, image generation, and fine-tuning. Released in October 2022, it shares the AD102 silicon family with the L40S but uses a different die revision and clock configuration. The two GPUs look similar on paper but perform differently in practice.
Takeaways
- 48GB GDDR6 at 864GB/s makes the L40 a strong single-GPU card for 7B-34B inference and image generation.
- The L40 delivers half the FP8 throughput of the L40S (362 vs 733 TFLOPS dense) because it lacks a Transformer Engine.
- No NVLink and no MIG rule the L40 out for distributed training and multi-tenant serving; the A100 and H100 fit those jobs.
- The A100 and H100 lead on memory bandwidth, but the L40 wins on tokens-per-dollar for single-GPU inference at low-to-moderate batch sizes.
- Thunder Compute rents the L40 at $0.79/hr with per-minute billing and no commitments.
NVIDIA L40 at a Glance
| Specification | Value |
|---|---|
| Architecture | Ada Lovelace (AD102-895) |
| CUDA Cores | 18,176 |
| Tensor Cores | 568 (4th Gen) |
| RT Cores | 142 (3rd Gen) |
| GPU Memory | 48GB GDDR6 ECC |
| Memory Bandwidth | 864GB/s |
| FP32 Performance | 90.5 TFLOPS |
| FP16 Tensor Performance | 181 TFLOPS (362 TFLOPS with sparsity) |
| FP8 Tensor Performance | 362 TFLOPS (724 TFLOPS with sparsity) |
| L2 Cache | 96 MB |
| Base / Boost Clock | 735 MHz / 2,490 MHz |
| TDP | 300 W |
| Form Factor | PCIe Gen4 x16, dual-slot, full-height full-length |
| NVLink | No |
| MIG Support | No |
| Launch Date | October 2022 |
| Source: NVIDIA L40 Datasheet (January 2023). | |
Rent an L40 on Thunder Compute, $0.79/hr with per-minute billing and no commitments.
Architecture: Ada Lovelace and the AD102 Die
The L40 is built on NVIDIA's Ada Lovelace architecture, manufactured on TSMC's 4N process using the AD102-895 die. The same die family powers the L40S (AD102-896) and the RTX 4090.
CUDA Cores, Tensor Cores, and RT Cores
The L40's 568 fourth-generation Tensor Cores support FP8, BF16, FP16, and TF32 precision, covering every major mixed-precision training and inference workflow. The card packs 18,176 CUDA cores across 142 streaming multiprocessors. Its 142 third-generation RT Cores handle hardware ray tracing, which is not relevant for AI compute workloads.
FP8 and the Transformer Engine
The L40 supports FP8 in hardware but does not include the L40S's Transformer Engine. The Transformer Engine automatically recasts between FP8 and BF16 at the layer level during inference and training. The L40S ships with it as a rated capability; the L40 exposes the FP8 datapath but leaves that dynamic mixed-precision management to the framework.
The L40 delivers 362 TFLOPS at FP8 versus the L40S's 733 TFLOPS. That gap is a hardware difference: although both cards use the AD102 die, they are different revisions (AD102-895 vs AD102-896), and the L40S runs at a much higher base clock and 350W versus the L40's 300W, roughly doubling its FP8 throughput. The Transformer Engine is a separate advantage that makes FP8 easier to deploy in frameworks, not the source of the raw TFLOPS gap.
For a full breakdown, see the L40 vs L40S comparison.
Memory Subsystem: 48GB GDDR6 and 864GB/s Bandwidth
The L40 uses 48GB of GDDR6 ECC memory across a 384-bit interface, delivering 864GB/s of bandwidth. The L40 and L40S share the same memory subsystem, so both hit 864GB/s. The A100 80GB outpaces the L40 on memory bandwidth: 2,039GB/s via HBM2e versus 864GB/s via GDDR6. That advantage is why the A100 leads in bandwidth-bound training despite having fewer CUDA cores.
Performance: What You Get from 90.5 TFLOPS FP32
FP32, FP16, and FP8 TFLOPS
The L40 delivers 90.5 TFLOPS at FP32, 181 TFLOPS at FP16 Tensor (362 with sparsity), and 362 TFLOPS at FP8 (724 with sparsity). For LLM serving, the figures that matter are FP16 and FP8 Tensor performance, since inference workloads rarely run at FP32. The L40S doubles both to 362 TFLOPS at FP16 and 733 TFLOPS at FP8.
LLM Inference Throughput
The L40 generates roughly 75-100 tokens/second on a 7B model at batch size 1 in FP16, and 15-25 tokens/second on a 70B model at INT4. At batch size 8, throughput on 7B models scales to 250-350 tokens/second in FP16. Actual throughput varies by quantization method, context length, and framework version.
Image Generation
The L40 handles Stable Diffusion XL within its 48GB with significant headroom. ComfyUI workflows with multiple ControlNet passes or video generation pipelines run on a single L40 without memory spilling, which is not always possible on 24GB cards.
Fine-Tuning
For LoRA and QLoRA fine-tuning of 7B-13B models, the L40's 48GB removes most memory bottlenecks at typical batch sizes. Full fine-tuning of models above 13B parameters in FP16 will exhaust the available memory. QLoRA with 4-bit quantization is the standard workaround for larger models.
What Models Fit in 48GB?
The table below shows approximate VRAM requirements for common models. Figures cover model weights only. KV cache, activations, and framework overhead add to these totals at runtime.
| Model | Precision | Approximate VRAM | |
|---|---|---|---|
| Llama 3 8B | FP16 | ~16GB | |
| Llama 3 8B | INT4 (QLoRA) | ~5GB | |
| DeepSeek R1 32B | Q4 | ~18GB | |
| DeepSeek R1 70B | Q4 | ~33GB | |
| Llama 3 70B | INT4 | ~35GB | |
| Flux.1 Dev | FP16 | ~24GB | |
| Stable Diffusion XL | FP16 | ~7GB | |
| Wan 2.2 T2V-14B | FP16 | ~28GB | |
| VRAM estimates reflect model weights only. KV cache, activations, and framework overhead add additional requirements at runtime, particularly at larger batch sizes and context lengths. | |||
How the L40 Compares: L40S, A100, and H100
The L40 stays competitive with the A100 and H100 for single-GPU inference and image generation at a fraction of the hourly cost, even though it sits below both on raw throughput and memory bandwidth. Its Ada Lovelace Tensor Cores and 48GB of VRAM carry that value. The table below places the L40 against the three GPUs.
| Specification | L40 | L40S | A100 | H100 SXM |
|---|---|---|---|---|
| Architecture | Ada Lovelace | Ada Lovelace | Ampere | Hopper |
| VRAM | 48GB GDDR6 | 48GB GDDR6 | 80GB HBM2e | 80GB HBM3 |
| Memory Bandwidth | 864GB/s | 864GB/s | 2,039GB/s | 3,350GB/s |
| FP16 Tensor (dense) | 181 TFLOPS | 362 TFLOPS | 312 TFLOPS | 989 TFLOPS |
| FP8 Tensor (dense) | 362 TFLOPS | 733 TFLOPS | Not supported | 1,979 TFLOPS |
| Transformer Engine | No | Yes | No | Yes |
| NVLink | No (PCIe Gen4) | No (PCIe Gen4) | Yes (600GB/s) | Yes (900GB/s) |
| MIG Support | No | No | Yes (up to 7) | Yes (up to 7) |
| TDP | 300W | 350W | 400W | 700W |
| Thunder Compute Price | $0.79/hr | ā | $1.09/hr | $3.20/hr |
Use Cases
AI Inference (7B to 70B Models)
The L40 serves 7B-13B models at FP16 and larger models at INT4. Its 48GB of VRAM holds Llama 3 70B at 4-bit quantization alongside a reasonable KV cache, making it a practical single-GPU inference server for most production API workloads. At batch sizes below 16, memory bandwidth is not the binding bottleneck, so the L40 delivers competitive tokens-per-dollar against higher-bandwidth GPUs.
Image and Video Generation
Pipelines with multiple ControlNet passes, IP-Adapter layers, or high-resolution upscaling benefit from the L40's memory headroom. Video generation models such as Wan and CogVideoX also fit within the L40's memory budget at standard resolutions. Loading full pipelines without offloading reduces latency on the L40 compared to a 24GB card.
Fine-Tuning and Small-Scale Training
LoRA and QLoRA fine-tuning of 7B-13B models is a natural fit. The 48GB capacity removes most memory bottlenecks at typical training batch sizes, and FP8 support reduces memory pressure during forward passes. Full fine-tuning of larger models requires quantization or a multi-GPU setup, where the A100 SXM with NVLink is the more appropriate choice.
Limitations
No NVLink: What This Means for Multi-GPU Scaling
The L40 uses PCIe Gen4, providing up to 64GB/s of bidirectional bandwidth. The A100 SXM adds NVLink at 600GB/s per GPU, and the H100 SXM scales that to 900GB/s per GPU with NVLink 4.0. For tensor parallelism across a multi-GPU training run, the L40's PCIe bottleneck becomes significant. The L40 works well in single-GPU deployments and horizontally scaled inference clusters, but not in tightly coupled distributed training.
No MIG Partitioning
The L40 does not support Multi-Instance GPU (MIG) partitioning. The full 48GB VRAM is allocated to a single workload per instance. For multi-tenant serving of many small models with hardware isolation, the A100 with MIG is the right tool.
Choosing the L40: Best-Fit Workloads
The L40 is a value-tier data center GPU, not a training flagship. It earns its place when 48GB of VRAM and Ada Lovelace inference performance matter more than raw memory bandwidth or multi-GPU scaling.
Choose the L40 when:
- You are serving 7B-34B models on a single GPU, where 864GB/s bandwidth is sufficient at low-to-moderate batch sizes and the L40's lower hourly rate wins on tokens-per-dollar.
- You run image or video generation pipelines (Flux, SDXL, ControlNet stacks, Wan) that benefit from 48GB of headroom to avoid CPU offloading.
- You are doing LoRA or QLoRA fine-tuning of 7B-13B models, where the 48GB capacity removes most memory pressure at typical batch sizes.
Choose a different GPU when:
- You need distributed training with tensor parallelism across multiple GPUs. The L40 has no NVLink, so step up to the A100 SXM or H100 SXM.
- You are running high-concurrency inference at batch size 16+, where the A100 or H100's memory bandwidth becomes the decisive factor.
- You need multi-tenant serving with hardware isolation, which requires MIG on the A100 or H100.
- FP8 throughput is your bottleneck. The L40S delivers roughly double the L40's dense FP8 performance and adds the Transformer Engine for automatic FP8 recasting.
NVIDIA L40 Cloud Pricing
Marketplace platforms offer lower starting rates but variable availability. Dedicated GPU cloud providers offer more predictable access with fixed pricing.
| Provider | L40 Price (per GPU/hr) | Notes |
|---|---|---|
| Thunder Compute | $0.79 | Per-minute billing, no commitments |
| Runpod | $0.99 | Marketplace, availability varies |
| Vast.ai | From $0.53 | Marketplace, host-dependent reliability |
| CoreWeave | $1.25 | 8-GPU node minimum |
| Pricing reflects public on-demand rates as of September 2026. Marketplace pricing fluctuates based on live availability. | ||
See the full L40 and L40S pricing guide for a complete provider breakdown.
Last Thoughts on the NVIDIA L40
The L40 is a practical choice for teams that need 48GB of VRAM for inference, image generation, or fine-tuning without paying for training-tier hardware. For higher inference throughput, the L40S is worth the premium. For distributed training with NVLink, the A100 SXM is the right tool. For most other workloads, the L40 at $0.79/hr on Thunder Compute delivers strong performance per dollar.
FAQ
How much VRAM does the NVIDIA L40 have?
The NVIDIA L40 has 48GB of GDDR6 ECC memory with 864GB/s of memory bandwidth. This is enough to run Llama 3 70B at INT4 quantization, Flux.1 Dev at FP16, and most image generation pipelines without CPU offloading.
Does the NVIDIA L40 support FP8?
Yes. The L40 reaches 362 TFLOPS at FP8 (724 TFLOPS with sparsity) via its fourth-generation Tensor Cores. Unlike the L40S, the Transformer Engine is not enabled by default, so it does not dynamically switch between FP8 and BF16 at the layer level during inference.
Does the NVIDIA L40 support NVLink or MIG?
No. The L40 supports neither NVLink nor MIG partitioning. Multi-GPU communication runs over PCIe Gen4. For workloads needing tight GPU-to-GPU interconnect or tenant isolation via MIG, the A100 SXM is the appropriate alternative.
Is the NVIDIA L40 good for LLM training?
The L40 is well-suited for LoRA and QLoRA fine-tuning of 7B-13B models on a single GPU. For large-scale distributed training above 13B parameters in full precision, or workloads that require NVLink, the A100 SXM is the better choice.
What is the L40's TDP?
The NVIDIA L40 has a 300W TDP, lower than the L40S (350W) and the A100 SXM (400W). This makes it more power-efficient in rack-dense deployments where thermal headroom is a constraint.
What models can I run on the NVIDIA L40?
The L40's 48GB fits Llama 3 8B at FP16 (16GB), Llama 3 70B at INT4 (35GB), Mistral NeMo 12B at FP16 (24GB), Flux.1 Dev at FP16 (24GB), and SDXL at FP16 (~7GB). Models requiring more than 48GB at full precision need quantization or multi-GPU setup.
Can the NVIDIA L40 handle distributed training?
Not efficiently. The L40 has no NVLink and communicates over PCIe Gen4 at 64GB/s, which bottlenecks tensor parallelism across GPUs. For distributed training at 70B+ scale, the A100 SXM or H100 SXM is the better choice.
How much does it cost to rent an NVIDIA L40?
Thunder Compute rents the NVIDIA L40 at $0.79/hr with per-minute billing and no commitments. Marketplace platforms list variable rates from around $0.53/hr, while dedicated providers like CoreWeave run higher at $1.25/hr with an 8-GPU node minimum.