Skip to content
Blogs

The 'I Am Legend' Effect in RL Environments

The agent is the last one left in the RL environment

Get started with AgentEnv Framework by following our docs.

The last man and the last agent

The 2007 I Am Legend poster: Will Smith and his dog walk alone along an empty waterfront below the ruined Brooklyn Bridge, under a golden sky

I Am Legend (2007)

The RL Environments (2026) poster: Claude Code's orange mascot stands alone in an overgrown, empty avenue at golden hour

RL Environments (2026)

Left: I Am Legend (2007), a post-apocalyptic movie where Will Smith battles loneliness (and zombies)Right: RL environments resemble “I Am Legend”

In a real company, work doesn't wait for you. Emails get answered, approvals get denied, and records change while a task is still in progress. Most RL environments remove all of this: the agent acts alone, and the world stays frozen between its moves, so agents trained there never learn to handle any of it. We call this the "I Am Legend" effect, after the Will Smith film in which he plays the last man in New York: every day at noon he broadcasts to the city, and for most of the film nobody answers.

In this blog post we will uncover how the "I Am Legend" effect underlies most RL environments and how the AgentEnv Framework provides you the tools for removing it.

1. Evals have the "I Am Legend" effect

Data is the main driver of model gains in 2026, yet most of the data used for training and evaluating frontier models isn't a realistic portrayal of the real world. We looked at the 20 agent evals cited most in 2026: every one of them has the "I Am Legend" effect, and only two, OSWorld 2.0 and τ³-bench, do anything to soften it.

RL Environment Benchmarks have the “I Am Legend” effect

The agent is the only one in its world: nobody replies, and nothing changes until it acts.

Who else is in the env:

the agent alone: nobody else in its envsomebody else, in some or all tasks

The 20 agent evals cited most in 2026, most cited first, and who else is in each one’s environment. Tap or click a benchmark to see why.

Frontier models have hill-climbed every eval so far in 2026, and will continue to do so. The long pole is no longer doing the work, but defining the outcome (i.e., the eval). It stands to reason that evals are literally setting the direction of frontier model improvement. Yet, in 2026 the evals are not representative of the real world, as we have shown above. They do not test agents in environments with dynamic events, where requirements shift, and cooperation is required.

2. Fill the environment with people

Getting any work done in our day-to-day jobs requires cooperating with other people. Evals today rarely include them: of the 20 above, only two put anyone else in the environment at all. They do not evaluate how well an agent communicates, collaborates, and coordinates with the people around it.

Take the example below: the agent is asked to resolve a Linear issue. Alone in the environment, the agent creates a PR that addresses the issue, and that's the end of it. In a realistic environment, the agent must create the PR and cooperate with a human engineer who reviews it, resolving their concerns to get the PR approved for merging. While doing so, a product manager edits the Linear issue, shifting the requirements for the agent's task.

One task, two environments

Turn 1 of 7

The task on both sides: ship ENG-142, “Export reports as CSV”

  1. 1Solver opens PR #42
  1. 1Solver opens PR #42

What the solver had to deal withalonewith others

Turns used
0vs0
Review comments
0vs0
Changes of plan
0vs0
Approvals
0vs0
The same task in the startup env, turn by turn, alone and with a reviewer and a PM.

Other people bring the entropy of real work into a task: a reviewer pushes back, a plan changes halfway through. For a task designer, that adds degrees of freedom in creating tasks, something we have found helpful at Scale.

In the AgentEnv Framework, each of these people is played by an agent. Each one is two task steps: a deploy_agent step that connects it to the same AgentEnv Environment as the agent being tested, and a prompt_agent step that gives it its job. Adding a person to a task adds those two steps, and the environment and the verifier stay as they are.

One task, any number of agents

The solver alone in startup, as in most benchmarks: the first game above.

Agent joinssolver· role solver+2 steps

Who
Claude Code, the agent being trained. Its prompt is the task: ship ENG-142.
Adds
Deploys solver into startup, the env every agent sharesdeploy_agentenv_ids: ["startup"], a2a_agent_id: "claude-code", agent_name: "solver", role: "solver"Gives it its jobprompt_agentagent_name: "solver", prompt_id: "ship", prompt: "Ship ENG-142: open a pull request and keep it up to date until it is approved."
Same
The env and the verifier don’t change:deploy_envstartuprubrics_verifiershipped
Add people to a task one agent at a time: the env step and the verifier never change.

3. Script the dynamics of the environment

The real world has entropy, which we want to include, but not at the cost of reliably evaluating the model in our environment. In the same way that Seinfeld looks like real life, but is scripted, we want our environment to look like real life, but be scripted in order to minimize entropy.

The AgentEnv Framework supports this with three kinds of triggers:

  • Environment triggers react to what happens in the environment. When an agent makes a certain tool call, or the environment reaches a certain state, the trigger calls a tool, turns a role's tools on or off, or hands a plain-language instruction to an agent that plays a person, such as a supplier or an auditor. Below, every refund held for approval posts a request to #finance-approvals, and opening a phishing link turns off every tool and alerts security.
  • Clock triggers do the same on the environment's virtual clock: at a set time, a fixed delay after another trigger, or on a schedule, even at random gaps that a seed keeps the same in every run. Merging switches off at 17:00 on Friday and back on at 09:00 on Monday.
  • Agent triggers script the people the agent talks to. Rules on a simulated user answer for it on a given turn, when the agent's reply matches, or once an environment trigger has fired. The simulated customer below pushes back from turn 3 and says thanks only once the refund has gone through.

These triggers can be combined into a graph of triggers that ultimately lets you be the director of your environment.

Twelve scenarios, scripted with triggers

At 10:15 the storefront’s checkout starts answering 503 and PagerDuty pages the on-call agent. Merging a revert heals it, and the merge is held until checkout is back.

Waiting to run
Pick a scenario to watch its triggers fire as the agent works.

4. Case study: the PortSim environment

PortSimEnv puts both ideas, other people and a scripted world, to work in one environment built on real data. The agent plans the berths on APM Terminals Barcelona's container quay for the real week of 12 February 2024, and the 7 shipping agencies that act for its 17 ships are each played by an agent it can message. The week is scripted: each time the environment clock advances into a new watch, a trigger fires and posts the port's news, from revised ship arrival times to wind forecasts. The agent must coordinate with the agencies to succeed: across PortSim's 15 weeks, the same scripted planner earns more than twice the reward when it asks them for arrival times (0.55) as when it never does (0.24).

PortSimEnv v5: a week at the Port of Barcelona

GPT-6 Astra plans APM Terminals Barcelona’s quay for the busy week of 12 February 2024, in the wind of a real storm. The 7 shipping agencies that act for its 17 ships are each an agent it can message, and the port’s news reaches it through triggers each time it advances the week.

Mon 12 Feb · 00:00

The weekMon 12 Feb · 00:00watch 0

Known
Terminal ops: Sections 10-18 are closed from hour 0 to hour 39 (fender repair).Barcelona Port Control: Wind forecast issued Sun 21:00 (ECMWF, adjusted to the Dique Sur anemometer), to hour 60: above 25 kn hours 55–61, peak 28 kn.A. Pérez y Cía: ZIM ATLANTIC (ship 0) calls in: at buoy S, arriving hour 1 (Mon 01:00).
Free of the “I Am Legend” effect:7 other agents40 triggered noticesa virtual clock
A week of PortSimEnv v5 as GPT-6 Astra played it, filmed from the run and captioned from its rollout. Watch the full replay on Hugging Face. Twin © OpenStreetMap contributors (ODbL); forecasts ECMWF open data (CC BY 4.0, modified); wind windows derived from Meteocat XEMA Y7.

2027 will bring evals where the agent is no longer the last one left, with people to work with and a world that changes around it. The AgentEnv Framework is proud to support this new kind of eval.

Try AgentEnv Framework

It's open source, and one install away. Its hello task runs on your machine with no Docker, model or cloud account.

pip install agentenv-framework

Then follow a cookbook to build a complete task, from an empty folder to a graded run.

Acknowledgements

Written by Edgar Arakelyan, Razvan Dumitru, Alina Trinh, and Kelvin Luu.