Skip to content
Blogs

The Harness Is the Player

A harness that kept rewriting itself let a small model keep up with a big one.

Get started with AgentEnv Framework by following our docs.

1. The question

If you let an LLM recursively improve its own harness, could it outperform a stronger model at the same tasks? We set out to answer that with three games: basketball, poker, and a cooking simulator.

Tl;dr: Claude Haiku 5.5 started well behind Claude Opus 5.5 in all three. After a handful of edits to its harness it was level in two of them and most of the way there in the third, at a fraction of the cost. The edits that mattered were beyond just improving prompts. The loop turned previous judgements into computation. Every change that persisted in the harness was something that Haiku was guessing at before.

Every replay in this post is a real game, recorded as it was played, so you can watch it happen.

2. The setup

Haiku 5.5 and Opus 5.5 start behind an identical thin harness:

  • the prompt
  • a renderer that turns game state into text
  • an action menu
  • a memory file
  • a few subroutines.

While Opus keeps the thin one, Haiku's gets iterated on.

The loop is simple. An external agent reads Haiku's recent games and proposes two edits to the harness. A critic throws out anything keyed to specific seeds or opponent quirks. Each candidate is played on a fixed set of games, and it is admitted only if Haiku plays measurably better than before, above a noise band we measured first. Then the admitted harness is tested on games nobody trained on. A generation is one admitted edit. The recipe is adapted from RRSI, with the modular split from ModularRSI and the open-world framing of Harness².

How one generation is made

One turn of the loop. A round can admit one edit or none, and only an admitted edit counts as a new generation.

3. Basketball

Opus coaches purple on the thin harness; Haiku coaches orange on its gen 4 harness. A full game to eleven from the final head-to-head set; Haiku wins 11–5. Each coach’s latest call sits beside the score. Toggle the switch to see the calls over the players’ heads; time slows for a beat whenever a coach makes a call.
Opus’s edge, pts per 100 possessions
+220
gen 0 → gens 1 to 4; within noise of zero from gen 1 on
Haiku’s wins vs Opus
31%41%
10 of 32 → 13 of 32, after the first admitted edit
Haiku turnovers per possession
29%16%
after the first admitted edit
Haiku vs a scripted team
−9+18
pts per 100 possessions on fresh games, gen 0 → gen 4
Before and after the harness edits, same 32 seeds per generation. Opus’s edge disappears after the first edit; Haiku turns the ball over half as often. A game to 11 is about 30 possessions.

Haiku and Opus each coach a team of three. The players are not scripted bots; instead each one is a fast decision model (OpenAI's Luna) that picks its own move every second of game clock from whatever its coach last said. Both coaches send instructions to their three player agents at the start of every play and at stoppages.

Then the loop's first admitted edit landed. It added three computed facts to what the coach sees:

  • passing lanes coverage
  • the value of each zone on the floor
  • a suggested play.

Haiku's win rate rose from 31% to 41%, and its turnovers fell from 29% to 16%. Over the next three generations the two coaches stayed even on points per possession. Opus still won 78 of 128 games across those generations, a record that leans Opus but does not clear the bar we set before the runs. Against a fixed scripted team on games it never played on, Haiku's gen 4 coach was clearly better than its gen 0 self.

Harness improvements

  • computational
  • prompt-level
  • computedGen 1: a per-option value line computed from the court state: which passing lanes are covered, the value of each zone, and a suggested play.
  • computedGen 2: memory that keeps factual tallies instead of impressions: points per possession by the play the team opened with, and each opponent's drives by side and result.
  • promptGen 2: notes on how to read those tallies: lean a defense only toward a side the tally shows repeated drives on, and never contest a dribbler.
  • computedGen 3: the value table recalibrated to observed shooting, so pull-up threes stop being overrated; a close defender discounts the pull-up and adds a pump fake; the decision point moves to the top of what the coach reads.
  • computedGen 4: the table tuned for possession starts and the moment after a drive beats its man.
  • promptGen 4: the coach's notes aligned with the new table, and the set play no longer treated as the free default. The same table without these notes was not admitted; with them it was.

4. Poker

Haiku on its evolved harness is on the left, Opus on the thin harness on the right. Two hundred recorded hands, with both hole hands and each model's reason as it acts.

Haiku vs Opus on the thin harness, big blinds per 100 hands (95% CI)

gen 0−168gen 3+26gen 6+35

Gen 0: 400 hands. Gens 3 and 6: 1,600 each. The other generations met Opus for 100 to 400 hands, too few to chart.

Haiku’s result against Opus, by generation. Opus stays on the thin harness throughout; only Haiku’s harness changes.

Poker is no limit heads-up, and every deal is played twice with the seats swapped so luck cancels. Haiku started at a strong disadvantage: −167.8 big blinds per 100 hands against Opus, and it lost every 100 hand session.

Within three generations of harness edits, Haiku was averaging +25.8 bb/100 against the same Opus. After six, it was averaging +35.1 bb/100. Over 1,600 hands we could no longer tell the two apart. All of that while costing about 94% less than Opus per 100 hands.

Harness improvements

  • computational
  • prompt-level
  • computedGen 1: a renderer line that names the hand, counts outs and compares pot odds to the equity needed to call.
  • promptGen 1: the action menu reworded so a slightly malformed reply is no longer silently dropped.
  • computedGen 2: a running tally of how often the opponent folds and calls.
  • promptGen 2: the strategy text keyed to the hand label the renderer now prints.
  • computedGen 3: a one-card chart of preflop ranges and standard responses.
  • computedGen 4: a first-in button line: open-raise or fold, never limp; is this hand in range, yes or no.
  • promptGen 4: raise_pot is the preflop value size.
  • computedGen 5: a postflop strength-tier line with a default action attached.
  • promptGen 5: bet bigger against callers, but only with strong hands.
  • computedGen 6: paired-board labels, so an overpair on a 9-5-5 board is no longer called two pair, and a session memory.

Haiku didn't lack any of this poker insight. However, it was getting a lot of this computation wrong under pressure.

5. The Kitchen

A tomato kitchen from the later rounds, same unseen start state, side by side. Each cook works its own half of the room, with different stations on each side. Left, Opus on the thin harness: 10 soups, 812 points. Right, Haiku on its final evolved harness: 6 soups, 660 points. Thin-harness Haiku managed 4 soups and 314 points on this start, so the harness closed about 70% of the gap here, in line with this kitchen's average. The strip above each kitchen shows the live orders as onion and tomato dots. Both clocks start together.

Soups delivered per game on unseen starts

Haiku · thin4.7Haiku · gen 27.4Haiku · gen 49.5Opus · thin11.6

50 unseen starts per arm. Gen 4 closes 70% of the gap [63, 77]; gens 5 to 10 closed 54% of a wider gap in harder kitchens.

Soups per game on start states Haiku had never seen, with Opus on the thin harness as the ceiling.

Each model gets two cooks and one kitchen. The model is the captain, giving both cooks orders, and racing to deliver as many meals as it can in 400 steps, each cook making one move per step, while new orders keep arriving.

This was where we saw the biggest gap. Opus beat thin-harness Haiku in 50 of 50 training rounds. On the split kitchen, where each cook can only reach half the stations, Haiku sent cooks to places they could not get to 1,786 times and ran out of its call budget in every episode.

The loop's fix was a plan scaffold: before any order goes out, the harness checks which stations each cook can actually reach and refuses the rest. After four admitted generations, on start states Haiku had never seen, it had closed 70% of the gap to Opus. On kitchens it had never seen at all it was nearly four soups ahead of its old self. It was still about two soups a game behind Opus.

Harness improvements

  • computational
  • prompt-level
  • computedGen 1: one renderer line per cook listing the stations it can reach and the counters both cooks can reach.
  • promptGen 1: target only reachable stations, relay through hand-off counters, start full pots at once.
  • computedGen 2: a next-target suggestion per cook, built from what it holds and what each pot needs, plus a per-pot line (needs two more, start it now).
  • promptGen 2: an empty hand at a pot starts a partial soup; pick up from a counter only once the item is there.
  • computedGen 3: relays for split kitchens: the supply cook stages ingredients and a dish on a hand-off counter, the pot-side cook carries them.
  • computedGen 4: pathing that treats the other cook as an obstacle and picks a station it can actually reach.
  • computedGens 5 to 7: a serve play; per-cook targets built only from reachable stations; a per-pot line of what the best open order still needs and whether starting it now pays zero.
  • computedGen 8: a distinct relay counter per cook, and a dish with no soup cooking gets parked on a counter only its holder can reach.
  • promptGen 8: goto works only on floor cells; when a cook cannot reach a station, items go through a counter both cooks touch.
  • computedGen 9: the cannot-reach list persists for the whole game instead of resetting each call.
  • promptGen 10: three rules and nothing else: an empty hand at a pot starts it, queue a counter pickup only when the item is listed there, never start a pot whose line says it pays zero. The biggest single gain of the lineage, 396 to 472 points on the training starts.

The later generations ran in harder kitchens: tomatoes as a second ingredient, three live orders, recipe values. There thin Haiku scored 232 points a game on unseen starts to Opus's 636. By gen 10 Haiku was at 451, 54% of the gap closed, and 178 points ahead of its thin self on kitchens it had never seen.

6. Running it yourself

  • Ask at decision points, not every second. Reflex cadence hid every difference between the models.
  • Expect code first, then prose about the code. The edits that transferred computed something; the prompt lines that stuck told the model how to use it.
  • Run the big model on the evolved harness too. It is the cheapest control and it changed our reading the most.
  • Don't let "cheaper" count as "better". When iterating on a harness, let the agent know that cost is a tie-breaker between approaches, but not the final goal.

  1. Environment

    arena, fixed seeds

  2. Agent artifact

    harness, gen N

  3. Paired runs

    same seeds, both sides

  4. Proposer + critic

    two edits, checked

  5. Admit?

    gain above noise

  6. ↺ the admitted edit becomes gen N+1, a new version of the same artifact
  7. every stage pre-registered with hashed files and a spend meter · every figure in this post replays one of these runs
The loop as it ran on AgentEnv Framework. The harness is an agent artifact; each admitted edit is a new version of it, so a lineage is just the artifact’s history.

All of it ran on AgentEnv Framework. Each harness generation is a versioned agent artifact, so a lineage is just a chain of them. Every comparison is paired: identical seeds, duplicate deals, the same start states. Each stage was pre-registered with hashed frozen files and a spend meter before a paid episode ran. And every replay in this post is a recorded trajectory, not an illustration.

Try AgentEnv Framework

It's open source, and one install away. Its hello task runs on your machine with no Docker, model or cloud account.

pip install agentenv-framework
Star on GitHub386 starsGet startedStar on GitHub

Then follow a cookbook to build a complete task, from an empty folder to a graded run.

Acknowledgements

Experiment, harnesses and write-up by Revan Aleti with the AgentEnv team.