How to Run Harbor Agent Evals in Docker Sandboxes MicroVMs

Summary

• Harbor, the open-source agent eval framework, can now run every eval task inside Docker Sandboxes
• Each task gets its own microVM with its own kernel, not a shared container, so untrusted agent code can’t reach the host or other tasks
• Docker Cloud Sandboxes add network policies that allow dependency downloads during setup, then cut internet access while the agent works
• The same task runs locally or in the cloud; Docker says it has tested 64 concurrent tasks in Cloud Sandboxes
• The integration is on the Docker Sandboxes nightly release now, with the stable release due October 23

Purpose

To evaluate a coding agent safely and fairly. An eval hands the agent a real task, lets it install packages, edit files and run commands, then checks the result, and a benchmark repeats that thousands of times. As Docker’s announcement points out, that is untrusted, AI-written code, and agents don’t always play fair: an agent can pass an eval by tampering with tests or overwriting the file that records its score, a behavior known as reward hacking. Running evals well needs two things, a framework that runs tasks and collects scores (Harbor) and a clean, isolated environment for every task (Docker Sandboxes).

Pre-requisites

• Python with uv installed, since Harbor installs as a uv tool
• A Docker account with Cloud access activated, for Cloud Sandboxes
• A Docker personal access token with Cloud Sandboxes selected under access permissions
• For local runs, the Docker Sandboxes CLI, per the Docker Sandboxes docs
• A benchmark or task set for Harbor, such as Terminal-Bench

Steps

1. Install Harbor with the Docker Sandboxes extra. Run uv tool install "harbor[sbx]". Until the stable release, the integration needs the nightly build: uv tool install --upgrade "harbor[sbx]==0.24.1.dev202610150414".

2. Create a Docker access token. Activate Cloud access on your Docker account, create a personal access token, and select Cloud Sandboxes under access permissions. The token is shown only once, so store it in your secrets manager straight away.

3. Export your credentials. Set SBX_DOCKER_USERNAME to your Docker ID and SBX_DOCKER_PAT to the token.

4. Run a smoke test in the cloud. harbor run -t hello-world/hello-world -a oracle -e sbx runs the hello-world task with Harbor’s oracle agent in a Cloud Sandbox. Harbor calls sandboxes “environments,” which is why the flag is -e; according to the Harbor sandbox docs, sbx runs in the cloud by default.

5. Switch to local when iterating. Install the Docker Sandboxes CLI, run sbx login, and add --ek target=local to the same command. Local runs currently support public and no-network modes, subject to the machine’s existing sandbox policy.

6. Lock down the network for fair runs. In Cloud Sandboxes, configure network policies so tasks download dependencies during setup and lose internet access while the agent works. Every agent then faces the same conditions.

7. Scale out. Use Harbor’s -n flag to set concurrent trials (the Harbor docs list a default of 4). Docker has tested 64 concurrent tasks and asks teams needing more to contact it.

8. Run multi-container tasks. With the stable release, Docker Compose support lets Harbor run services and sidecars together inside a single Cloud Sandbox microVM.

The Trap

Treating a container as a sandbox. Harbor’s default environment is still plain docker, a container that shares the host kernel, and a shared kernel is the boundary a reward-hacking or misbehaving agent might probe. Docker’s case for microVM isolation is that each task gets its own kernel, so what the agent does stays inside its sandbox. Also note that local mode is limited to public and no-network modes for now; reproducibility claims should be based on Cloud Sandbox runs with an explicit network policy. For a wider view of the options, see Stack Archive’s comparison of cloud agent sandboxes across AWS, Azure, Google Cloud and Cloudflare and the LangSmith team’s agent eval framework.

Cited References

• Docker: Harbor + Docker Sandboxes: Run Safe, Reproducible Agent Evals
• Harbor docs: Pre-integrated sandboxes
• Docker Sandboxes documentation
• Docker: Why MicroVMs, the architecture behind Docker Sandboxes

Source: Docker Blog

Content created with AI assistance and reviewed for accuracy.