To get started with cloud infrastructure, it's essential to understand its nomenclature. Terminology can vary depending on the provider, however the terms listed below are industry standards.
This guide is a cloud computing 101, breaking down key terms you'll encounter when working with modern GPU clouds.
Cloud Computing Terms
| Term | Definition | Further Reading |
|---|---|---|
| Bare Metal | A physical server dedicated to a single tenant with no virtualization layer, giving direct access to the underlying hardware for maximum performance and predictable throughput. | |
| Computer Vision | A field of AI that enables machines to interpret and analyze images and video. | Read more |
| Egress Fees | Charges for transferring data out of a cloud provider’s network. | Read more |
| GPU-as-a-Service (GPUaaS) | Cloud model where GPUs are rented on demand for compute workloads. | Read more |
| Hyperscaler | Large cloud providers offering massive global infrastructure (e.g., AWS, Azure, Google Cloud). | |
| Instance | A virtual machine running in the cloud with allocated compute resources. | |
| LLM Inference | The process of running a trained large language model to generate outputs. | |
| Neocloud | A newer category of GPU cloud providers focused on AI workloads. | Read more |
| On Demand Instances | Instances billed per second or minute with no long-term commitment. | Read more |
| Persistent Storage | Storage that survives instance stops, restarts, and terminations. Unlike ephemeral disk attached to a running instance, persistent storage retains data between sessions and can be remounted on a new instance. | |
| Reserved Instances | Instances purchased at a discount in advance and for a certain period. | |
| Snapshot | A point-in-time copy of an instance or volume that captures the full environment, including installed packages, model weights, and configuration. Used to save progress, clone environments, or resume work on a different instance type. | |
| Spot Instances | Discounted instances using spare capacity, which can be interrupted with little notice. | Read more |
Graphic Processing Unit (GPU) Concepts
| Term | Definition | Further Reading |
|---|---|---|
| CUDA Cores | Programmable processing units in NVIDIA GPUs used for general computation. | Read more |
| DGX | NVIDIA's line of purpose-built AI supercomputing systems that integrate multiple GPUs, high-speed interconnects, and optimized software into a single turnkey platform. | Read more |
| FP8 / Mixed Precision | Numerical formats that reduce the bit-width of model weights and activations during training or inference. FP8 (8-bit float) is supported natively by the H100's Transformer Engine and delivers up to 3–4x the throughput of FP16 on transformer workloads while maintaining accuracy. | Read more |
| HBM (High Bandwidth Memory) | A stacked memory architecture used in data center GPUs that delivers significantly higher bandwidth than standard GDDR memory. Variants include HBM2e (A100) and HBM3 (H100 SXM), with higher generations offering greater bandwidth for memory-bound workloads. | Read more |
| HGX | NVIDIA's reference server board platform that allows OEM partners to build GPU-accelerated servers by integrating multiple high-end GPUs with NVLink and NVSwitch. | Read more |
| InfiniBand | A high-speed network interconnect used to link multiple GPU nodes for distributed training. Delivers low latency and high bandwidth (typically 400 Gbps or more) needed for efficient gradient synchronization across nodes. | Read more |
| MIG (Multi-Instance GPU) | An NVIDIA feature that partitions a single GPU into up to seven hardware-isolated instances, each with dedicated CUDA cores, Tensor Cores, and HBM memory. Useful for running multiple smaller models concurrently on one physical GPU. | Read more |
| NVLink | NVIDIA's high-bandwidth, low-latency GPU-to-GPU interconnect that enables faster data sharing between GPUs than PCIe allows. | Read more |
| NVSwitch | NVIDIA's high-speed switch chip that enables full all-to-all NVLink connectivity across multiple GPUs within a node or across nodes. | Read more |
| PCIe | A standard high-speed interface used to connect GPUs, storage, and other peripherals to a computer's CPU and motherboard. | Read more |
| ROCm | AMD's open-source software platform for GPU computing, providing an alternative to NVIDIA's CUDA ecosystem for running AI and HPC workloads on AMD GPUs. | Read more |
| Shannon Entropy | A mathematical measure of the average uncertainty, randomness, or information content inherent in a data source or probability distribution. | Read more |
| SXM | NVIDIA's high-bandwidth socket form factor for mounting data center GPUs directly onto a baseboard, enabling faster NVLink connections compared to standard PCIe cards. | Read more |
| Taiwan Semiconductor Manufacturing Company (TSMC) | The world's largest semiconductor producer, controlling nearly 70% of the advanced microchips global market. | Read more |
| Tensor Cores | Specialized GPU cores optimized for AI and matrix operations. | Read more |
| Transformer Engine | A dedicated hardware and software component in NVIDIA GPUs that was first introduced with the Hopper architecture (H100). It accelerates transformer-based AI models by automatically switching between FP8 and FP16 precision. | Read more |
| VRAM (Video RAM) | Dedicated high-speed memory on a GPU used to store model weights, activations, and intermediate tensors during computation. VRAM capacity is typically the primary constraint when deciding which models can run on a given GPU. |
AI Concepts
| Term | Definition | Further Reading |
|---|---|---|
| Hidden Dimension | The width of the vectors a model uses to represent information internally between layers, often called the hidden size or d_model in transformers. Larger hidden dimensions increase representational capacity along with parameter count and VRAM footprint. | |
| Latency | The time elapsed between submitting a request to a model and receiving the first token of its response, typically measured in milliseconds. Lower latency is critical for real-time, interactive applications. | |
| Mixture of Experts (MoE) | An architecture that routes inputs to specialized sub-networks ("experts"), activating only a fraction of total parameters at once to scale capacity efficiently. | |
| Model | A machine learning algorithm trained on data to recognize patterns, make predictions, or generate outputs. | |
| Model Context Protocol (MCP) | An open standard that allows AI models to securely interface with external data sources, developer tools, and client environments. | Read more |
| Open-source model | An AI model whose source code, core architecture, and training pipelines are entirely public and freely modification-accessible. | Read more |
| Open-weights model | A model whose pre-trained numerical weights are publicly accessible for deployment, though its exact training data or source code may remain proprietary. | Read more |
| Parameter | The internal numerical variables (weights and biases) that a model optimizes during training to define its behavior and overall capacity. | |
| Quantization | A compression technique that converts model weights from high-precision formats (like FP16) to lower-bit structures (like FP8 or INT4) to save VRAM. | Read more |
| RAG (Retrieval-Augmented Generation) | An inference architecture that augments a model's response by retrieving relevant documents from an external knowledge base at query time. Allows models to answer questions about data they were not trained on without requiring fine-tuning. | |
| Throughput | The number of tokens a model generates per second across all concurrent requests, indicating how efficiently an inference system utilizes hardware under load. | |
| vLLM | An open-source inference engine optimized for serving large language models at scale. Implements PagedAttention for efficient KV cache management and continuous batching to maximize GPU utilization across concurrent requests. | Read more |
AI Training
| Term | Definition | Further Reading |
|---|---|---|
| Activation | The intermediate outputs each layer produces during the forward pass. They are held in memory to compute gradients during the backward pass, making them one of the largest consumers of VRAM during training. | Read more |
| Backward Pass | The stage of training where the model computes gradients of the loss with respect to each weight and propagates them back through the layers, a process known as backpropagation. It is usually the most memory- and compute-intensive part of a training step because it depends on the activations saved during the forward pass. | Read more |
| Batch Size | The number of samples processed together in a single forward and backward pass. Larger batch sizes improve GPU utilization and throughput but require proportionally more VRAM to store activations. | Read more |
| Checkpoint | A saved copy of a model's weights and optimizer state at a point during training. Used to resume interrupted runs, evaluate intermediate model quality, or roll back to an earlier training state. | |
| Distributed Training | A training strategy that splits a model, dataset, or both across multiple GPUs or nodes to handle workloads that exceed the capacity or throughput of a single GPU. Requires high-bandwidth interconnects such as NVLink or InfiniBand to synchronize gradients efficiently. | Read more |
| Forward Pass | The stage where input data flows through the model's layers to produce an output. During training, the activations computed at each layer are kept in memory so they can be reused when calculating gradients in the backward pass. | |
| Gradients | The partial derivatives of the loss with respect to each model weight, computed during the backward pass and used to update parameters. In distributed training, gradients are synchronized across GPUs every step, which makes interconnect bandwidth a common bottleneck. | Read more |
| LoRA (Low-Rank Adaptation) | A parameter-efficient fine-tuning technique that inserts small trainable weight matrices into a frozen base model instead of updating all parameters. Dramatically reduces VRAM requirements and training time compared to full fine-tuning. | Read more |
| RLHF (Reinforcement Learning from Human Feedback) | A training technique that uses human preference data to optimize a model beyond what supervised fine-tuning alone can achieve. Typically applied after SFT to improve helpfulness, accuracy, and safety through a reward model trained on human rankings. | Read more |
| Supervised Fine-Tuning (SFT) | Training a model on labeled data to improve task-specific performance. | Read more |
Takeaway
This glossary is just the starting point. For deeper dives into pricing, infrastructure, and performance comparisons, explore the Thunder Compute blog.
By building a strong foundation in cloud computing terminology, you'll be better equipped to navigate the rapidly evolving landscape of GPU cloud infrastructure.