A reinforcement learning (RL) environment is the system agents act in while a model learns from its results. It hands the agent a task, applies the agent's actions and returns a reward. RL environments for LLM agents run each attempt at a task (also called "episodes" or "rollouts") in an isolated sandbox.
For LLM agents, the environment is usually a codebase, a browser or a set of tools. The agent's actions execute code and commands, so running thousands of episodes safely is an infrastructure problem. Sandbox isolation, GPU placement, preemption and pricing determine the speed and cost of RL training runs.
Takeaways
- Many RL environments only need CPUs; reward models, local inference, GPU simulators and GPU code tasks require a GPU.
- Published agent RL setups run 128-512 rollouts in parallel, mostly in CPU sandboxes, with the model served on separate GPUs.
- Preemption hurts RL more than most workloads, because an evicted rollout usually restarts from scratch.
- 100 steps of 128 GPU rollouts cost about $6,830-$11,290 on H100 sandboxes.
What Is an RL Environment?
An RL environment is code that defines the task an agent practices on. It controls four things:
- Starting state: how each episode begins
- Observations and actions: what the agent can see and do, and how its actions change the world
- Reward: how the result is scored
- Termination: when the episode ends
State, Actions, Rewards and Episodes
Every RL environment runs the same loop:
- The agent receives a state or observation.
- It picks an action.
- The environment returns a new state and a reward.
One full run from the starting state to the end is an episode. The stream of episodes is what training learns from.
Classical RL formalizes this loop as a Markov decision process. Early machine learning benchmarks such as CartPole and Atari games made the loop easy to see, because their states and actions were small and simulated.
RL Environments for LLMs and Agents
RL environments for LLMs are built from real software, and the agent's actions are real commands and clicks. A coding environment gives the agent a repository, a shell and failing tests, and the agent edits files and runs commands. A browser environment shows the agent screenshots and lets it click and type in a live page. The reward is often verifiable: the tests pass, the answer is correct or the task is complete.
Because those actions execute for real, each episode needs an isolated sandbox. The agent installs packages and runs commands that an untrusted model chose, and the sandbox keeps them from damaging the host or other episodes. It also gives every episode the same clean starting state, which keeps rewards comparable across the batch.
What LLM Agent Rollouts Need From a Sandbox
Classic RL environments focusing on games need no isolation. The agent sends joystick inputs to an emulator, and a reset call restores the starting state. LLM agent rollouts are different: the agent's actions execute code and commands, and each episode changes files, packages and processes that no reset call can undo. That puts four requirements on the sandbox.
One Sandbox per Episode
A fresh sandbox per episode gives every rollout an identical starting state. Files written, packages installed and processes started in one episode can't leak into the next, which keeps rewards comparable across the batch. Destroying the sandbox is the reset.
Containing Untrusted Actions
Mid-training, a policy produces plenty of bad actions: infinite loops, deleted directories and network calls to places it shouldn't reach. A sandbox contains them. Research on sandbox infrastructure for agentic training lists untrustworthy agent execution as a core problem that rollout infrastructure has to solve.
Handling Bursty Sandbox Creation
Demand for sandboxes comes in waves. A batch of episodes starts together, so hundreds of sandboxes are needed at once, and then demand drops while the trainer updates the model. That makes start time and concurrency limits as important as the hourly price.
Keeping Long-Lived Sessions Alive
An episode isn't a single request. The agent runs many commands over minutes, and the sandbox holds its files, packages and processes the whole time. Losing the sandbox mid-episode loses the whole attempt.
When an RL Environment Needs a GPU
RL training for LLMs always uses GPUs, but they usually sit outside the sandbox. Trainer GPUs update the model, and generation GPUs serve it to many rollouts through a shared endpoint. The sandbox typically runs only the environment: the repository, the shell and the tests. A GPU belongs inside the sandbox only when the environment itself needs one.
Reward Models and Learned Verifiers
Learned reward models are a common reason to put a GPU in the sandbox. A reward model that scores each trajectory needs a GPU, and running it inside the environment keeps scoring next to the agent's output.
In-Environment Inference and Simulators
Agents that call a local model during the episode need a GPU in the sandbox. So do environments that wrap a physics or rendering simulator. Most coding tasks graded by unit tests run on CPUs, and keeping them on CPU sandboxes keeps large fleets affordable.
Tasks that grade GPU code need a GPU in the environment too. For example, CUDA Agent trains an agent with RL to write CUDA kernels. Its environment compiles kernels in a CPU Docker sandbox, then sends verification and profiling jobs to a dedicated GPU sandbox pool of 128 NVIDIA H20 GPUs. The paper credits process-level isolation and exclusive resource allocation for avoiding interference between jobs.
Running Thousands of Rollouts in Parallel
Agent RL runs rollouts in large parallel batches. SWE-MiniSandbox (a container-free RL framework for software engineering agents) trains with a batch size of 16 and 8 rollouts each. That setup runs 128 isolated environments per update.
NVIDIA's NeMo Gym SWE RL case study runs 512 concurrent task instances: 16 prompts with 32 rollouts each. Its software engineering steps take at least 20 minutes for training and rollouts combined.
Neither setup gives each sandbox its own GPU. The sandboxes are CPU-bound, and GPUs train the model and serve it to every rollout from outside the sandboxes.
RL episodes also need time limits. In the RL stage of Microsoft's Orchard framework, each task gets a 10-minute wall-clock budget instead of a step limit. Orchard aborts rollouts that exceed the budget and leaves them out of training. A rollout budget also caps how long any one sandbox lives.
Matching Rollout Throughput to the Trainer
Rollout throughput has to match the trainer. When sandboxes produce trajectories more slowly than the trainer consumes them, training GPUs sit idle.
Long-tail episodes make matching harder. The WAR paper on workload-aware rollouts notes that the longest rollout in a batch can run many times longer than the median. Workers with short rollouts then wait at the end of each step.
Timeouts limit long-tail rollouts. A sandbox lifetime that matches the episode budget stops a stuck rollout from holding a GPU for the rest of the step.
Why Preemption Breaks RL Rollouts
Preemption costs RL more than most workloads, because an evicted rollout loses everything since the episode began.
Many GPU sandboxes can be preempted. Modal's GPU Sandboxes are subject to preemption, and Daytona's spot GPU sandboxes can be terminated without notice. Every eviction adds a rerun and delays its training step.
Read nextRead our guide to GPU sandboxes and how GPU isolation works.
What GPU Rollouts Cost at Scale
If each of 128 parallel rollouts needs its own GPU, a training step costs $68-$113 on H100 sandboxes. That scenario borrows the scale of SWE-MiniSandbox's 128 environments per update, which ran on CPUs. With rollouts capped at 10 minutes, one step uses around 21 GPU-hours, and 100 steps use 2,133 GPU-hours.
The table prices that workload on an H100 sandbox with 8 vCPUs, 32 GiB of memory and 100 GiB of disk at list prices.
| Option | GPU-Hour | Step (128 Rollouts) | 100 Steps |
|---|---|---|---|
| Thunder Compute | $3.88 | $83 | $8,280 |
| Daytona spot | $3.20 | $681 | $6,8301 |
| Daytona on-demand | $4.88 | $104 | $10,410 |
| Modal | $5.29 | $1131 | $11,2901 |
| 1 Before evictions. Every evicted rollout adds the cost of a rerun. | |||
These rollout costs are an upper bound for one configuration. Rollouts that finish early, CPU-only environments and smaller GPUs all cut the total.
Read nextGPU Sandbox Pricing Full rate card and billing rules.
Running RL Environments on Thunder Compute
Thunder Compute's GPU sandboxes meet RL environment requirements. Each sandbox runs in its own Firecracker microVM (a lightweight, hardware-isolated VM) with a GPU.
Non-Preemptible H100 and A100 Sandboxes
Thunder Compute sandboxes are never preempted, so a rollout that starts finishes at the quoted price. H100 sandboxes start at $2.95/hr and A100 80GB sandboxes at $1.95/hr, which suits lighter reward models. Thunder Compute charges no region multipliers or egress fees.
A Fresh Sandbox for Every Episode, With Durable Jobs
Thunder Compute sandboxes are ephemeral, which matches the one-sandbox-per-episode pattern. Each rollout starts clean and leaves nothing behind. By default, commands run as durable jobs that survive client disconnects. Each sandbox's lifetime can be set to match the episode budget.
Request access to GPU sandboxes for RL environments on Thunder Compute.
Last Thoughts on RL Environments on GPUs
RL environments define what an agent can learn, and sandboxes decide how safely and cheaply that learning runs. Reward models, local inference and simulators put a GPU inside the sandbox, where preemption and per-second pricing shape the budget.
Match sandbox lifetimes to the episode budget, keep CPU-only environments off GPUs and choose capacity that won't evict a rollout halfway through.
FAQ
What Is an RL Environment?
An RL environment is the system an agent interacts with during reinforcement learning. It defines the starting state, the available actions, how actions change the world and how each episode is scored.
What Is the Difference Between an RL Environment and a Sandbox?
The RL environment defines the task, observations, actions and rewards. The sandbox is the isolated machine where one episode of that environment runs, usually one sandbox per rollout.
Why Do RL Rollouts Run in Sandboxes?
RL rollouts run in sandboxes because agent actions are untrusted and every episode needs a clean start. A sandbox contains harmful commands, and destroying it after each episode resets the environment.
Do RL Environments Need GPUs?
Only some do. An RL environment needs a GPU when a model runs inside it, such as a reward model, a local model the agent calls or a GPU simulator. Most coding environments graded by unit tests run on CPUs.
How Much Do GPU RL Rollouts Cost?
At 128 rollouts capped at 10 minutes, one training step uses around 21 GPU-hours. On an H100 sandbox, 100 steps cost about $6,830-$11,290 at list prices, before evictions on preemptible options.