Reproducible Environments with Virtual Clocks

Get started with AgentEnv Framework by following our docs.

RL training frontier models on environments demands reproducibility. Reproducibility is the capability of an environment to be deployed in the exact same state, whether you deploy it yesterday, today, or 10 years from now, and to change only in ways you control, such as the passing of time. Agents will still act differently from run to run; the environment shouldn't be why.

In our experience building RL environments at Scale AI, uncontrolled time is one of the biggest reasons an environment isn't reproducible. In this blog we will cover the failure modes in uncontrolled time and the efforts we take to control time when building our environments.

1. Problems with uncontrolled timeCopy Anchor Link

An RL task is only good insofar as it is realistic, difficult, and fair. Fair implies a task is solvable -- without controlling time a task can drift from solvable today to unsolvable tomorrow. In the example shown below: we deploy the engineering startup environment from our RBAC post today, tomorrow, and 10 years from now and it drifts from being solvable today to unsolvable simply by the passage of time in the real world. The root cause: the environment uses the sandbox's machine clock -- as it changes with the passage of time so does the solvability of the task.

2. Ways to control timeCopy Anchor Link

Once you realize your environment has to control time, the obvious approach comes to mind first: set the clock of the machine the environment runs on, the OS kernel's system clock that every program on it asks in the end. Dig a little deeper and you discover that most programs ask the kernel through the C library (clock_gettime(), gettimeofday()), where a shim like faketime can step in and answer with the date you choose. Higher up the stack still, you can pin the time in the application itself: a clock of the environment's own that its servers ask in their code. The layers for controlling time are shown below:

AgentEnv controls time at the application layer for three reasons. You can't always control the kernel: a container shares it with its host. The lower layers never reach the agent, which runs in a sandbox of its own. And AgentEnv's open protocol makes the application layer simple: the gateway holds the clock, one task step arms it, and each server asks it with a few lines of code.

3. Controlling time with AgentEnv's virtual clockCopy Anchor Link

Every AgentEnv Environment gateway holds a virtual clock as the single source of truth on what time it is, and the servers in the environment use the virtual clock through AgentEnv's open protocol.

To use the gateway's clock in your environment, give each server a sync_time extension under urn:agentenv:clock/v1. When a task arms the clock, the server is handed the gateway's time URL, and from then on it reads the time there instead of from its machine, as this email server does when it stamps the emails it sends:

from datetime import datetime, timezone

import httpx
from agentenv_protocol import AgentEnvEnvironment, extension, tool


class EmailEnv(AgentEnvEnvironment):
    ...

    @extension(uri="urn:agentenv:clock/v1",
               description="Sync this server to the gateway's clock.")
    async def sync_time(self, env_get_time_url: str) -> dict:
        self.env_get_time_url = env_get_time_url
        return {"now": await self._now()}

    async def _now(self) -> str:
        if self.env_get_time_url:
            async with httpx.AsyncClient() as client:
                resp = await client.get(self.env_get_time_url, timeout=10)
            if resp.status_code == 200:
                return resp.json()["virtual_time"]
        return datetime.now(timezone.utc).isoformat()

    @tool()
    async def send(self, to: str, body: str) -> dict:
        """Send an email, stamped with the environment's time."""
        self.sent.append({"to": to, "body": body,
                          "sent_at": await self._now()})
        return {"sent": True}

It is important for agents and the environment to agree on what time it is. The gateway exposes a get_time tool. Agents use the get_time tool to get the same time the servers do. A single source of truth for what time it is for both the agent and the environment.

With AgentEnv Framework it is very simple to integrate the virtual clock into your task. Simply add the sync_env_clock task step as the task DAG below shows:

{"id": "clock","type": "sync_env_clock","env_id": "startup","virtual_time": "2026-10-15T08:00:00Z","virtual_seconds_per_real_second": 60,"depends_on": [{"task_step_id": "assistant"}]}

Realistic tools and data are only part of a reproducible environment; the other part is controlling time. Without it, a task that is fair today can be unsolvable tomorrow, and scores measure the calendar instead of the agent. Controlling time isn't easy, AgentEnv Framework makes it easy.