Can Agents Run an Airline More Efficiently Through a Storm?

Get started with AgentEnv Framework by following our docs.

AirOpsEnv reconstructs 22 JetBlue storm days from 2023–2026. An ops agent manages flights and aircraft, a crew agent assigns pilots and flight attendants, and a customer agent tracks connections and stranded passengers. Each has only its role's tools, enforced through modify_env_tool_access (more on role permissions). To hold a flight for connecting passengers, the customer agent has to ask ops.

Why realism mattersCopy Anchor Link

JetBlue attributed $50M in lost revenue and $35M in lost operating income to 4,100 cancellations in the first quarter of 2014. We wanted to test whether agents could reduce losses like those. That requires a simulator where keeping a flight operating means finding capacity, a gate, an aircraft and a legal crew.

Flights come from Bureau of Transportation Statistics records, FAA ground stops and delay programs retain their issue times, and weather comes from airport reports, forecasts and NEXRAD radar. Crews and passengers are generated from FAA duty rules and DOT passenger data. Hourly airport capacity and gate space are limited using recorded traffic. Lightning slows ramp operations at JFK and Boston; aircraft can break down, swaps are allowed only at those two hubs, and the fleet must finish in position for tomorrow's departures.

The agents start at 04:00 with about 700 flights, 220 aircraft and 900 crews and reserves. They work in hourly watches, seeing flight delays two hours ahead and airport capacity as it stands now. AgentEnv's virtual clock pauses the day while they think.

We checked the simulator by replaying JetBlue's recorded decisions. On the nine held-out days, daily delay totals correlated 0.98 with the record, and total delay was 3.6% higher. None of these days was used for fitting, so the match means the simulator's capacity, weather and crew limits behave like the real ones: looser or tighter limits would push JetBlue's own decisions far from the record.

Plans earn higher rewards for lower estimated net airline cost: aircraft delay, crew moves and hotels, aircraft repositioning, late starts the next morning, and lost fares, minus fuel and maintenance saved by cancelling. We fitted the share of stranded passengers' fares lost and the savings from cancelling on 13 days against JetBlue's 2014 disclosure, then tested on the other nine. Passenger time and the cost of running the AI aren't included. Every run must pass an audit of information timing, permissions and crew legality; all of Astra's did.

Results on the nine held-out daysCopy Anchor Link

GPT-6 Astra's plans cost $18.91M, compared with $20.16M for scripted desks following standard airline practice and $26.02M for the JetBlue replay. That's a 6% saving against the scripted desks and 27% against the replay. The scripted desks are the cleaner comparison: they operate in the same simulator, while JetBlue's controllers had information absent from the public record. Astra's advantage over the scripted desks came from less departure delay.

Astra beat the scripted desks on seven days, missing our preregistered target of eight. Its two losses were 1.3% and 0.9%. The pooled saving held when we varied the lost-fare share from 50% to 100% and cancellation savings from zero to the full fuel and maintenance cost.

Astra left 169 flights unflown, compared with JetBlue's 528 recorded cancellations. In the passenger simulation, about 16,000 people were stranded overnight under Astra's plans, versus 58,000 under the replay: 1.9 million fewer hours of delay at final destinations, mostly overnight waits. Passengers on flights that operated were about as late as JetBlue's. Against the scripted desks, passenger delay fell 2.5%.

Where the model failsCopy Anchor Link

Astra's confirmed changes carried $4.2M in projected savings; its measured gain over the scripted desks was $1.25M. On July 9, 2023, ops cancelled two round trips claiming the move “net saves $112k with no new autos”. Replaying the day without those four cancellations cost $8,000 less.

On June 23, 2024, ops cancelled the Martha's Vineyard–Boston flight at 13:00 to free an aircraft for Raleigh-Durham, then cancelled that flight too at 23:00. Seven of its cancellations were flights the scripted desks flew; five were cancelled after their scheduled departure.

Three runs of June 27, 2023, cost between $5.04M and $5.37M, a spread as large as several daily margins. The best hindsight plan cost another $4.5M less, mostly through lower delay and end-of-day crew costs. It cancelled 139 more flights than Astra.

How agents could help controllersCopy Anchor Link

We'd start with an agent reviewing proposed cancellations and crew moves. Ops rejected most cancellation requests from the other desks, and accepting them all would have cost another $510,000. A controller could ask an agent to examine a proposed move across the network, show which flights and passengers it affects, and suggest alternatives before committing.

The controller would make the final call using the crew and aircraft information available on the day.

Run a realistic dayCopy Anchor Link

You'll need Docker, uv and git, and a model endpoint and key that serve the model you pass to --model. Install the plugin and build the environment and agents:

uv tool install agentenv-framework \
    --with "agentenv-airops @ git+https://github.com/earakely-scale/agentenv-airops-plugin@4ebd72663d0e7cc7f7b71a141ffb2c2dfde3c2c7"
agent-env airops-occ3 setup

Generate the 22 days and run one with three agents:

agent-env airops-occ3 tasks --realistic --split all
agent-env run results/bundles/airops-occ3-real --task b6-20230627-0400-joint-real --model openai/gpt-6.1-sol