The 'I Am Legend' Effect in RL Environments
The agent is the last one left in the RL environment
Get started with AgentEnv Framework by following our docs.
The last man and the last agent

I Am Legend (2007)

RL Environments (2026)
In a real company, work doesn't wait for you. Emails get answered, approvals get denied, and records change while a task is still in progress. Most RL environments remove all of this: the agent acts alone, and the world stays frozen between its moves, so agents trained there never learn to handle any of it. We call this the "I Am Legend" effect, after the Will Smith film in which he plays the last man in New York: every day at noon he broadcasts to the city, and for most of the film nobody answers.
In this blog post we will uncover how the "I Am Legend" effect underlies most RL environments and how the AgentEnv Framework provides you the tools for removing it.
1. Evals have the "I Am Legend" effect
Data is the main driver of model gains in 2026, yet most of the data used for training and evaluating frontier models isn't a realistic portrayal of the real world. We looked at the 20 agent evals cited most in 2026: every one of them has the "I Am Legend" effect, and only two, OSWorld 2.0 and τ³-bench, do anything to soften it.
RL Environment Benchmarks have the “I Am Legend” effect
The agent is the only one in its world: nobody replies, and nothing changes until it acts.
the agent alone: nobody else in its envsomebody else, in some or all tasks
Frontier models have hill-climbed every eval so far in 2026, and will continue to do so. The long pole is no longer doing the work, but defining the outcome (i.e., the eval). It stands to reason that evals are literally setting the direction of frontier model improvement. Yet, in 2026 the evals are not representative of the real world, as we have shown above. They do not test agents in environments with dynamic events, where requirements shift, and cooperation is required.
2. Fill the environment with people
Getting any work done in our day-to-day jobs requires cooperating with other people. Evals today rarely include them: of the 20 above, only two put anyone else in the environment at all. They do not evaluate how well an agent communicates, collaborates, and coordinates with the people around it.
Take the example below: the agent is asked to resolve a Linear issue. Alone in the environment, the agent creates a PR that addresses the issue, and that's the end of it. In a realistic environment, the agent must create the PR and cooperate with a human engineer who reviews it, resolving their concerns to get the PR approved for merging. While doing so, a product manager edits the Linear issue, shifting the requirements for the agent's task.
One task, two environments
Turn 1 of 7
The task on both sides: ship ENG-142, “Export reports as CSV”
- 1Solver opens PR #42
- 1Solver opens PR #42
What the solver had to deal withalonewith others
- Turns used
- 0vs0
- Review comments
- 0vs0
- Changes of plan
- 0vs0
- Approvals
- 0vs0
Other people bring the entropy of real work into a task: a reviewer pushes back, a plan changes halfway through. For a task designer, that adds degrees of freedom in creating tasks, something we have found helpful at Scale.
In the AgentEnv Framework, each of these people is played by an agent. Each one is two task steps: a deploy_agent step that connects it to the same AgentEnv Environment as the agent being tested, and a prompt_agent step that gives it its job. Adding a person to a task adds those two steps, and the environment and the verifier stay as they are.
One task, any number of agents
The solver alone in startup, as in most benchmarks: the first game above.
Agent joinssolver· role solver+2 steps
- Who
- Claude Code, the agent being trained. Its prompt is the task: ship ENG-142.
- Adds
- Deploys solver into
startup, the env every agent sharesdeploy_agentenv_ids: ["startup"], a2a_agent_id: "claude-code", agent_name: "solver", role: "solver"Gives it its jobprompt_agentagent_name: "solver", prompt_id: "ship", prompt: "Ship ENG-142: open a pull request and keep it up to date until it is approved." - Same
- The env and the verifier don’t change:deploy_env
startuprubrics_verifiershipped
3. Script the dynamics of the environment
The real world has entropy, which we want to include, but not at the cost of reliably evaluating the model in our environment. In the same way that Seinfeld looks like real life, but is scripted, we want our environment to look like real life, but be scripted in order to minimize entropy.
The AgentEnv Framework supports this with three kinds of triggers:
- Environment triggers react to what happens in the environment. When an agent makes a certain tool call, or the environment reaches a certain state, the trigger calls a tool, turns a role's tools on or off, or hands a plain-language instruction to an agent that plays a person, such as a supplier or an auditor. Below, every refund held for approval posts a request to #finance-approvals, and opening a phishing link turns off every tool and alerts security.
- Clock triggers do the same on the environment's virtual clock: at a set time, a fixed delay after another trigger, or on a schedule, even at random gaps that a seed keeps the same in every run. Merging switches off at 17:00 on Friday and back on at 09:00 on Monday.
- Agent triggers script the people the agent talks to. Rules on a simulated user answer for it on a given turn, when the agent's reply matches, or once an environment trigger has fired. The simulated customer below pushes back from turn 3 and says thanks only once the refund has gone through.
These triggers can be combined into a graph of triggers that ultimately lets you be the director of your environment.
Twelve scenarios, scripted with triggers
At 10:15 the storefront’s checkout starts answering 503 and PagerDuty pages the on-call agent. Merging a revert heals it, and the merge is held until checkout is back.
4. Case study: the PortSim environment
PortSimEnv puts both ideas, other people and a scripted world, to work in one environment built on real data. The agent plans the berths on APM Terminals Barcelona's container quay for the real week of 12 February 2024, and the 7 shipping agencies that act for its 17 ships are each played by an agent it can message. The week is scripted: each time the environment clock advances into a new watch, a trigger fires and posts the port's news, from revised ship arrival times to wind forecasts. The agent must coordinate with the agencies to succeed: across PortSim's 15 weeks, the same scripted planner earns more than twice the reward when it asks them for arrival times (0.55) as when it never does (0.24).
PortSimEnv v5: a week at the Port of Barcelona
GPT-6 Astra plans APM Terminals Barcelona’s quay for the busy week of 12 February 2024, in the wind of a real storm. The 7 shipping agencies that act for its 17 ships are each an agent it can message, and the port’s news reaches it through triggers each time it advances the week.
Mon 12 Feb · 00:00· watch 0
The weekMon 12 Feb · 00:00watch 0
- Known
- Terminal ops: Sections 10-18 are closed from hour 0 to hour 39 (fender repair).Barcelona Port Control: Wind forecast issued Sun 21:00 (ECMWF, adjusted to the Dique Sur anemometer), to hour 60: above 25 kn hours 55–61, peak 28 kn.A. Pérez y Cía: ZIM ATLANTIC (ship 0) calls in: at buoy S, arriving hour 1 (Mon 01:00).
2027 will bring evals where the agent is no longer the last one left, with people to work with and a world that changes around it. The AgentEnv Framework is proud to support this new kind of eval.
Try AgentEnv Framework
It's open source, and one install away. Its hello task runs on your machine with no Docker, model or cloud account.
pip install agentenv-frameworkThen follow a cookbook to build a complete task, from an empty folder to a graded run.
Acknowledgements
Written by Edgar Arakelyan, Razvan Dumitru, Alina Trinh, and Kelvin Luu.