Coding-agent evals run benchmark tasks in isolated sandboxes. They do so in parallel to cut a full run from days to hours. These coding-agents edit real repositories and execute commands, making isolation a security requirement.
Many eval guides assume the model sits behind an API. But in some cases, the model being tested or used to judge results needs a GPU within the sandbox. This changes implementation and costs to address parallelism, preemption and required resources.
Takeaways
- Each eval task runs in its own sandbox to contain agent actions and keep results reproducible.
- Parallelism speeds up the process: 32 parallel runtimes cut a full SWE-bench Lite run to several hours.
- Some evals need a GPU in the sandbox when using self-hosted, fine-tuned or judge models.
- A 300-task SWE-bench Lite run costs about $160-$265 on H100 sandboxes.
Benchmark figures and provider details in this guide were verified in October 2026.
A Brief History of Coding-Agent Benchmarks
Coding benchmarks have moved from single functions to whole repositories. Along the way, the infrastructure they need also changed: function-level benchmarks needed little more than a process per problem, while repository-level tasks each need a full sandbox.
Function-Level Benchmarks: CodeXGLUE, APPS, HumanEval and MBPP
2021 was a big year for code generation benchmarks:
- CodeXGLUE came first, in February, with 10 tasks across 14 datasets for code understanding and generation.
- APPS followed in May with 10,000 Python problems.
- HumanEval arrived in July, with 164 Python problems, each with a function signature, a docstring and unit tests. OpenAI introduced it alongside Codex, together with the pass@k metric: the probability that at least one of k generated samples passes all the tests.
- MBPP followed in August with 974 entry-level problems written by crowd workers, each with three test cases.
Stronger Tests and Fresh Problems: HumanEval+ and LiveCodeBench
As models improved, two weaknesses of these benchmarks surfaced: thin tests and solutions in training data.
EvalPlus improved tests in 2023. HumanEval averages about 10 tests per problem, while its HumanEval+ extension averages about 764. Across 26 models, pass@k dropped by 19%-29%, and some model rankings changed.
LiveCodeBench fixed widely available solutions in 2024. This eval collects new problems over time from LeetCode, AtCoder and CodeForces, so scores can be limited to problems published after a model's training data ends. It also tests self-repair and predicting a program's output, not only writing code.
Both still grade one program at a time against tests, so they put little demand on sandbox infrastructure.
SWE-bench
SWE-bench moved evaluation from functions to repositories. All 2,294 tasks pair a real GitHub issue with tests. The agent reads the codebase, edits files and produces a patch.
Two SWE-bench subsets became common for coding-agent evaluation:
- SWE-bench Lite has 300 tasks, which makes it cheaper to run.
- SWE-bench Verified has 500 tasks screened by human annotators for clear issue descriptions and fair tests.
In February 2026, OpenAI stopped evaluating SWE-bench Verified, because an audit of 138 problems that o3 didn't consistently solve found that 59.4% had material test or description issues. In addition, every frontier model tested could reproduce the original fix or verbatim problem details for some tasks.
OpenAI recommends SWE-bench Pro instead. This is a separate benchmark with new tasks rather than a subset of SWE-bench, and has 1,865 tasks from 41 repositories. Its held-out and commercial splits aren't public, which makes training-data contamination less likely.
Why Coding-Agent Evals Run in Sandboxes
Coding-agent evals run in sandboxes for two reasons:
- Safety: The agent executes untrusted commands which could damage the machine without a sandbox.
- Reproducibility: Every task starts under the same conditions, and a previous task's installed packages or leftover processes can't change another's result.
Read nextRead our guide to GPU sandboxes and how GPU isolation works.
When Evals Need a GPU Inside the Sandbox
Coding-agent evals need a GPU in the sandbox only when a model runs inside the eval environment. When an API serves the model being tested, the sandbox runs only the repository, the shell and the tests. And CPUs can handle that workload.
Evaluating Self-Hosted or Fine-Tuned Models
Self-hosted and fine-tuned models need a GPU wherever the eval serves them. Placing the model in the same sandbox as the task keeps each eval self-contained. The model, the repository and the tests start together and disappear together, which suits models that fit on a single GPU.
Judge and Reward Models
Judge and reward models need a GPU when an eval scores output with a model instead of, or alongside, unit tests. A judge model rates code quality, and a reward model scores trajectories. The same reward models appear in RL environments for agent training, so eval and RL infrastructure often look alike.
Evaluating GPU Code Such as CUDA Kernels
Evals that grade CUDA code have to run that code, so each task sandbox needs a GPU. KernelBench gives a model 250 PyTorch tasks and asks it to write a faster CUDA kernel for each one. The eval then runs the kernel on the GPU to check that its output matches the PyTorch reference and to time it against that reference.
The kernel can compile on CPU. But timing is only reliable when the kernel has a dedicated GPU to itself.
Running Evals in Parallel
Parallelism is the main way to speed up coding-agent eval. OpenHands reports that a single SWE-bench Lite task can take about 10 minutes, so sequentially the 300-tasks take around two days to run. Running 32 tasks in parallel cut this down to several hours.
That speedup is why eval tooling is now built around parallelism:
- CoreWeave's SWE-bench guide runs each instance in its own sandbox, and its configurable worker count sets how many instances run at once.
- SWE-ReX (the SWE-agent team's remote execution framework) runs agent sessions across remote sandbox backends in parallel.
Setting Concurrency and Timeouts
Eval concurrency is capped by capacity, quota and budget, not by the benchmark. Most harnesses expose a worker count. The right worker count is the most sandboxes a team can run without hitting capacity errors or overloading a shared model server.
Per-task timeouts matter as much as concurrency. A timeout stops a stuck agent from holding a sandbox, and its resources, for the rest of the run. Set the timeout slightly above the typical task length, and record timeouts as failures, not silent drops.
Why Preemption Skews Eval Results
Preemption adds noise to coding-agent eval results. A sandbox reclaimed mid-task must be configured to restart the task adding cost and delay, and untracked failures quietly lower the pass rate.
Several GPU sandbox options can be preempted. Modal's GPU Sandboxes are subject to preemption, and Daytona's spot GPU sandboxes can also be terminated without notice. Evals on preemptible capacity should log every retry, so a reported score reflects the model rather than the infrastructure.
Request access to non-preemptible GPU sandboxes.
What a SWE-bench Eval Run Costs
A full SWE-bench Lite run costs about $160-$265 on H100 GPU sandboxes at list prices. At 300 tasks and about 10 minutes each, one run uses roughly 50 GPU-hours. The table prices that run on an H100 sandbox with 8 vCPUs, 32 GiB of memory and 100 GiB of disk.
| Option | Per Hour | SWE-bench Lite Run (50 GPU-Hours) | 10 Runs |
|---|---|---|---|
| Daytona spot | $3.20 | $1601 | $1,6001 |
| Thunder Compute | $3.88 | $194 | $1,940 |
| Daytona on-demand | $4.88 | $244 | $2,440 |
| Modal | $5.29 | $2651 | $2,6501 |
| 1 Before evictions which add the cost of a rerun. | |||
These eval costs assume a GPU in every task sandbox for the full 10 minutes. Tasks that finish early, a shared inference server or a smaller GPU all lower the total.
Read nextSee the full rate card and billing rules in our GPU sandbox pricing comparison.
Running Coding-Agent Evals on Thunder Compute
Thunder Compute's GPU sandboxes provide ephemeral, API-created environments running in their own Firecracker microVM (a lightweight, hardware-isolated VM) with a GPU attached.
Thunder Compute sandboxes are never preempted, so a task that starts finishes, and pass rates carry no hidden retries. H100 sandboxes cost $2.95/hr, $1/hr below Daytona's on-demand H100. A100 80GB sandboxes cost $1.95/hr for smaller models under test, with no region multipliers or egress fees.
By default, Thunder Compute commands run as durable jobs that survive client disconnects. A dropped connection on the machine driving the eval doesn't end the run. Each sandbox has one GPU, which fits models that run on a single 80 GB card; larger models under test belong on a shared inference server.
Read next
Last Thoughts on Coding-Agent Evals
Coding-agent evals have moved from single-function benchmarks like HumanEval to repository-level tasks like SWE-bench, which introduced the need for sandboxes. One sandbox per task keeps results clean, and parallelism turns a multi-day run into an afternoon.
A GPU belongs in the sandbox only when a model runs there; when it does, pick a layout, cap every task with a timeout and run on capacity that won't reclaim a task halfway through.
FAQ
What Is HumanEval?
HumanEval is a benchmark of 164 hand-written Python programming problems with unit tests, introduced by OpenAI in 2021. It scores models with pass@k, the probability that at least one of k generated solutions passes every test.
What Is SWE-bench Lite?
SWE-bench Lite is a 300-task subset of SWE-bench chosen to be cheaper to run. SWE-bench Verified is a separate 500-task subset that human annotators screened for clear issue descriptions and fair tests.
Why Run Coding-Agent Evals in Sandboxes?
Sandboxes keep coding agents from damaging the host, because agents execute commands the model chose. A fresh sandbox per task also keeps results independent, since nothing one task changes can affect the next.
Do Coding-Agent Evals Need a GPU Sandbox?
Coding-agent evals need a GPU Sandbox only when a model runs inside the eval environment, such as a self-hosted or fine-tuned model under test, or a judge or reward model. Evals against an API-served model only need CPU sandboxes.
How Much Does a Coding-Agent Eval Run Cost?
A 300-task SWE-bench Lite run uses roughly 50 GPU-hours at about 10 minutes per task. On an H100 sandbox, that costs about $194 on Thunder Compute, $244 on Daytona on-demand and $265 on Modal.