# Miles (https://www.agentenvframework.com/docs/training/miles)

> Train using Miles

## How a rollout runs

1. Miles opens a session for a sample of the split’s JSON Lines and calls `agent.run` with its URL and the row, which opens a session on the bundle’s OpenEnv server and starts the row’s task.
2. The server starts the task as an episode: it deploys the env, loads the task, and opens the agent’s turn, while SGLang writes the model’s first reply through Miles’ session, which records it.
3. SGLang parses the reply’s tool calls, and `run` calls each on the episode and sends its answer back as a tool message, turn after turn up to the row’s `max_turns`.
4. A reply without a tool call ends the turn it left open; a task with another turn answers `end_turn` with the next prompt. The task’s verifier scores the episode.
5. `run` puts the score in the sample’s metadata for `agent.reward`, and Miles trains on the turns its session recorded; a rollout the env failed leaves its group out. The episode is torn down.

Parts of the scene:

- **Miles**: Miles’ trainer generates each rollout through its agentic generate path, on SGLang, calling the agent function `--custom-agent-function-path` names with a session on its session server, which turns the recorded chat into the tokens it trains on. Miles owns the model, the generation and the update; the environment owns the reward.
- **A dataset row**: A line of `<split>.jsonl`: the task’s prompt as chat `messages`, and in `metadata` the `split`, `index` and `task_id` the agent function plays, the row’s tools, and the server’s settings. The prompt is yours to change; the task stays the same.
- **agent.run**: Miles’ agent function, called once per sample: it posts the conversation and the row’s tools to the session’s chat endpoint, calls each tool call SGLang parsed on the episode, and answers a call whose arguments aren’t a JSON object with why. It imports nothing of Miles.
- **The session**: One OpenEnv WebSocket session per rollout, waiting out a server at its limit or restarting, within `reset_timeout`. A rollout the env fails isn’t played again, since Miles’ session already holds the model’s turns.
- **The OpenEnv server**: `agent_env.openenv.build_app(bundle)` serves every task of a bundle as an OpenEnv env, the split `all` plus one per eval. With `token=`, it can run on another machine than Miles’.
- **The env**: The task’s env, deployed fresh for this episode in a sandbox, here a container on your Docker. Its tools are what the model calls.
- **The agent’s turn**: The task prompts one agent, and your model plays it from outside: Miles’ session server renders each request with the model’s chat template and the row’s tools, and SGLang parses the reply’s tool calls, which go to the episode.
- **The verifier**: The task’s own verifier step scores the episode once the turn ends, the same score an evaluation run records. With several scores, `reward=` picks the one to train on.
- **The reward**: `agent.reward`, Miles’ reward hook, reads the episode’s reward from the sample’s metadata. An episode the env failed, or that ended without a reward, is flagged, and `agent.keep_scored_groups`, Miles’ dynamic-sampling filter, leaves its group out of the step.

[Miles](https://github.com/radixark/miles), RadixArk's RL framework built from slime, plays agentic
rollouts through an agent function. For each sample, it opens a session on its session server, which
records every chat request made through it on SGLang, and calls a function you name with the
session's URL and the row. AgentEnv's agent function plays the row's task as one episode on a
bundle's OpenEnv server, the model chatting through that session. Four pieces connect Miles to a
bundle:

| Piece                                         | What it does                                                                                                                                                                                |
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `agent_env.openenv.build_app(bundle)`         | Serves every task of a bundle as an [OpenEnv](https://www.agentenvframework.com/docs/openenv.md) env: the split `all`, plus one split per eval.                                                                                 |
| `python -m agent_env.training.miles`          | Writes the bundle's rows as Miles' prompt data, a JSON Lines file per split: each row's chat messages, and in its metadata the task, its tools and the server's settings.                   |
| `agent_env.training.miles.agent.run`          | The agent function (`--custom-agent-function-path`): plays the row's episode, the model calling the row's tools through the sample's session, and returns the episode's reward.             |
| `agent.reward` and `agent.keep_scored_groups` | The reward hook (`--custom-rm-path`), which reads that reward, and the dynamic-sampling filter (`--dynamic-sampling-filter-path`), which leaves out the group of an episode the env failed. |

Miles' tokenization, SGLang's tool-call parsing and the training stay Miles' own: the session server
turns each sample's recorded chat into the tokens Miles trains on, and the task's verifier scores the
episode. The helper imports nothing of Miles.

## Install

Miles runs on Linux with NVIDIA GPUs. This page's run used four L40S, training and SGLang sharing
them. Run Miles in its image, and install the framework into its Python:

```bash title="Terminal"
docker run -d --name miles --gpus all --network host --ipc host --shm-size 64g --ulimit memlock=-1 \
  -v "$PWD:$PWD" -w "$PWD" radixark/miles:dev-202611020756 sleep infinity
docker exec miles pip install 'agentenv-framework[miles]>=0.9.1418'
```

`pip check` then lists some of the image's own pins as unmet; this page's run trained with them. The
bundle's server runs where its envs deploy, with Docker running, in an environment of its own:

```bash title="Terminal"
pip install 'agentenv-framework[openenv]>=0.9.1418'
```

## Train on GridSim

GridSim is the bundle the [TRL page](https://www.agentenvframework.com/docs/training/trl.md#train-on-gridsim) trains on. Each task runs
one grid battery for one day in the ERCOT day-ahead market: the model reads the day's 24 hourly
prices and submits a schedule, one integer an hour, to charge, sell or idle. The env grades the
schedule against the day's exact optimum: 0 for nothing useful, 0.2 for idling all day, 1 for the
optimum. The bundle has 2,000 train days and 200 eval days.

### Serve the bundle

```bash title="Terminal"
hf download earakely-scale/GridSimEnv-AgentEnv --repo-type dataset --include "bundles/gridsim-v1/*" --local-dir .
```

```python title="serve.py"
import uvicorn

from agent_env.openenv import build_app

app = build_app("bundles/gridsim-v1", max_concurrent_envs=64)
uvicorn.run(app, host="127.0.0.1", port=8200)
```

`max_concurrent_envs` covers a step's 32 rollouts. A rollout that finds the server full waits for
room, within `reset_timeout`.

### Write the rows

```bash title="Terminal"
python -m agent_env.training.miles http://127.0.0.1:8200 data
```

```text title="Output"
wrote all.jsonl, eval.jsonl, train.jsonl to data
```

The command plays one task of each toolset to list the tools its rows offer, so it deploys an env
once. Each row holds the task's system prompt and first prompt as chat `messages`, and in `metadata`
the split, index and task id the agent function plays, the row's tools, and the server it plays on
with its settings. Since the rows carry them, Miles' rollout processes need no settings of their own.

| Option            | Default | What it sets                                                                                                                         |
| ----------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `--max-turns`     | 4       | The model turns a rollout may take; the episode is ended after the last.                                                             |
| `--tool-timeout`  | 60      | One tool call's deadline in seconds; the server ends a call that misses it, and the episode.                                         |
| `--reset-timeout` | 600     | How long an episode may take to deploy, and each turn to be ended, in seconds; also how long a rollout waits for room on the server. |
| `--model-timeout` | 600     | How long the agent function waits for each of the model's replies, in seconds.                                                       |
| `--token-var`     | none    | The environment variable holding the server's token: see [Envs on another machine](#envs-on-another-machine).                        |

Each chat request carries the row's tools, so Miles' session server renders them with the model's
chat template as `tools=` would anywhere, and SGLang parses the model's calls.

### Warm start with SFT

Played as it comes, Qwen3-4B idles: it scores 0.18 on the 200 eval days. Warm-start it with the
TRL page's [`sft.py`](https://www.agentenvframework.com/docs/training/trl.md#warm-start-with-sft), in an environment with TRL, so it saves
`qwen3-4b-gridsim-sft`. This page's run trained it in Miles' container on one of the L40S, with `trl`
and `peft` installed there.

### Train with GRPO

This run differs from the other trainers' in one way: Miles' FSDP backend has no LoRA, so it is a
full fine-tune, at learning rate 1e-6 constant, across the four GPUs with SGLang colocated. The rest
is theirs: the same SFT warm start, 32 rollouts a step (4 days × 8) at temperature 1.1, thinking
off, and at most four model turns a rollout. The parts that are about AgentEnv:

```bash title="miles_gridsim.sh (excerpt)"
python3 /root/miles/train.py \
  --prompt-data data/train.jsonl --eval-prompt-data eval data/eval.jsonl \
  --input-key messages --metadata-key metadata \
  --apply-chat-template-kwargs '{"enable_thinking": false}' \
  --use-session-server --tito-model qwen3 --sglang-tool-call-parser qwen25 \
  --custom-generate-function-path miles.rollout.generate_hub.agentic_tool_call.generate \
  --custom-agent-function-path agent_env.training.miles.agent.run \
  --custom-rm-path agent_env.training.miles.agent.reward \
  --dynamic-sampling-filter-path agent_env.training.miles.agent.keep_scored_groups \
  ...
```

The whole script, as it ran, is [`miles_gridsim.sh`](/training/miles_gridsim.sh). Run it in Miles'
container from the folder holding `qwen3-4b-gridsim-sft` and `data/`. Miles' launcher refuses GPUs it
has no verified profile for, as it did the L40S, so the script starts Ray and submits `train.py` as
the launcher does. It evaluates on the 200 eval days before the first step and every 10 steps, one
rollout a day at the training temperature.

### What it learns

| Qwen3-4B | eval reward (200 days) |
|---|---|
| after SFT (GRPO step 0) | 0.536 |
| GRPO step 10 | 0.652 |
| GRPO step 20 | 0.695 |

Mean train reward of each GRPO step's 32 rollouts, by step: 1: 0.759, 2: 0.680, 3: 0.820, 4: 0.730, 5: 0.545, 6: 0.635, 7: 0.580, 8: 0.803, 9: 0.832, 10: 0.665, 11: 0.702, 12: 0.458, 13: 0.825, 14: 0.766, 15: 0.715, 16: 0.486, 17: 0.915, 18: 0.755, 19: 0.727, 20: 0.631.

| Qwen3-4B, eval reward on the 200 eval days | Miles   | SkyRL | verl | prime-rl | NeMo RL | TRL  | Unsloth |
| ------------------------------------------ | ------- | ----- | ---- | -------- | ------- | ---- | ------- |
| After SFT (GRPO step 0)                    | 0.54    | 0.54  | 0.52 | 0.56     | 0.55    | 0.57 | 0.55    |
| After 20 GRPO steps                        | 0.70    | 0.55  | 0.59 | 0.58     | 0.68    | 0.56 | 0.52    |
| After 40 GRPO steps                        | not run | 0.61  | 0.60 | 0.57     | 0.67    | 0.61 | 0.59    |

All start from the same SFT model and score it alike, within what one rollout a day can tell apart
(about ±0.05). Through Miles the eval reward rose from 0.54 to 0.65 after 10 steps and 0.70 after
20, most of it on the 4-hour batteries, the hardest days, from 0.33 to 0.60. The share of schedules
that scored 0 fell from 12% to 4%. GridSim's greedy rule scores 0.76 on these days.

The train reward, each step's mean over its 4 days, stays near 0.70 from the first step: the 80
days the run drew hold half the eval days' share of 4-hour batteries. Miles' run is a full
fine-tune and the others LoRA, and each is one run, so read the gaps as pointers, not a ranking.

A step took about 125 seconds: 61 generating the 32 rollouts and about 50 training. Of the 1,240
episodes played, 3 failed in the env and their groups were left out, and 4 tool calls had arguments
that weren't a JSON object, which the model was told.

## Settings Miles needs

| Setting                         | Value here                   | What it does                                                                                                                                                    |
| ------------------------------- | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--use-session-server`          | on (v1)                      | The session server that records each sample's chat; Miles' agentic generate path refuses to run without it. `v2`, tree serving, works too.                      |
| `--tito-model`                  | `qwen3`                      | The model family the session server tokenizes the recorded chat as.                                                                                             |
| `--sglang-tool-call-parser`     | `qwen25`                     | SGLang's tool-call parser for the model's chat template. Without it, Qwen3's calls stay in the reply's text, the episode sees none, and every rollout scores 0. |
| `--input-key`, `--metadata-key` | `messages`, `metadata`       | Where the rows hold the chat and what the agent function plays.                                                                                                 |
| `--apply-chat-template-kwargs`  | `{"enable_thinking": false}` | The template settings the model trains with, here as its SFT data was.                                                                                          |

## Envs on another machine

Miles calls the agent function in its rollout processes, and each opens its session on the URL in
its row, so the bundle's server must be one those processes reach. Give it a token, which every
request and WebSocket but `/health` must then carry:

```python title="serve.py"
import os

import uvicorn

from agent_env.openenv import build_app

app = build_app("bundles/gridsim-v1", max_concurrent_envs=64, token=os.environ["AGENTENV_TOKEN"])
uvicorn.run(app, host="0.0.0.0", port=8200)
```

```bash title="Terminal"
python -m agent_env.training.miles http://envs.internal:8200 data --token-var AGENTENV_TOKEN
```

`--token-var` names the environment variable that holds the token: the writer reads it, and the rows
record its name, never the token. The agent function reads it in each rollout process, so set it in
the environment Miles' Ray workers start in. Without a token, anyone who reaches the server deploys
and drives its envs.

The server speaks plain HTTP and WebSockets, so the token crosses the network unencrypted. Keep the
server on a private network that only Miles' nodes reach, or put a TLS proxy in front of it and give
the writer its `https://` URL, which the agent function's sessions open as `wss://`.

## Failed rollouts

A rollout isn't played again when the env fails it (a deploy, a call past its deadline, a server
that went away), when the model misses `model_timeout`, or when its episode ends without a reward:
Miles' session already holds the turns the model took. The agent function flags it, and
`keep_scored_groups` leaves its whole group out of the step, and Miles draws more rows until the
step's batch is full. Without the filter (`--dynamic-sampling-filter-path`), a flagged rollout trains
as a reward of 0. The filter also leaves out two groups the agent function can't flag:

* Miles aborts a sample whose session recorded no model call, as when the session's first request
  fails.
* A sample can come back without a `reward`, as when the agent function's child process died
  mid-episode. Miles keeps the turns its session recorded but has no metadata to merge.

In an eval, Miles runs no filter, so a flagged rollout scores 0.

Opening its session, a rollout waits for a server at its limit or one restarting, within
`reset_timeout`, so a restart delays rollouts rather than failing them.

A tool call whose arguments aren't a JSON object isn't a failure: the agent function answers it with
why, as the model's error, and the episode goes on. The GPU run showed why. Counted as env failures,
such calls were left out of training unpenalized, and within six steps the policy drifted into making
them.

Miles counts a filter's drops by reason, as `rollout/dynamic_filter/drop_<reason>`, only when the
filter returns Miles' own `FilterOutput`. `keep_scored_groups` returns a bool, since the helper imports
nothing of Miles, so its drops don't show there. `rollout/raw_reward_unfiltered` also includes the
flagged rollouts' zeros. To count the drops, wrap the filter in one of your own and pass it to
`--dynamic-sampling-filter-path` instead:

```python title="my_filters.py"
from miles.rollout.filter_hub.base_types import FilterOutput

from agent_env.training.miles.agent import keep_scored_groups


def keep_and_count(args, samples, **kwargs):
    if keep_scored_groups(args, samples, **kwargs):
        return FilterOutput(keep=True)
    return FilterOutput(keep=False, reason="agentenv_failed")
```

With `--dynamic-sampling-filter-path my_filters.keep_and_count`, Miles logs each step's dropped groups
as `rollout/dynamic_filter/drop_agentenv_failed`.

## What the agent function returns

Miles merges what `agent.run` returns into each sample's metadata, where the reward hook, the filter
and any of your own hooks read it:

| Key                      | What it holds                                                                                               |
| ------------------------ | ----------------------------------------------------------------------------------------------------------- |
| `reward`                 | The episode's reward, or 0 for a flagged rollout.                                                           |
| `agentenv_scores`        | Each score the task's verifiers recorded, by the id of the step that recorded it. The reward combines them. |
| `agentenv_tool_calls`    | The tool calls run on the episode.                                                                          |
| `agentenv_invalid_calls` | The tool calls whose arguments weren't a JSON object, answered with why.                                    |
| `agentenv_failure`       | Why the rollout was flagged, when it was.                                                                   |

## Multi-turn tasks

A task with several turns plays as one rollout. Each row offers `end_turn` with the task's tools: the
model calls it with its answer, and its result is the next turn's prompt, which the agent function
returns like any tool answer. When the model replies without calling a tool, its reply ends the turn
it left open, and any turns after it end empty. Tell the model in the system prompt to end each turn
with `end_turn`. A task that records several scores needs a reward that combines them, given to
`build_app`, here their mean:

```python title="serve.py"
app = build_app("bundles/my-bundle", reward=lambda scores: sum(scores.values()) / len(scores),
                max_concurrent_envs=64)
```

## What's supported

|                               |                                                                                                                                                                                                                                                                                                         |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Miles                         | v0.1.3 and main, with the session server bare (v1) or `v2`. The page's run used `radixark/miles:dev-202611020756`, Miles main at 7154ff8, with v1. A CI job plays rows through Miles' rollout, session server and router, with a stand-in SGLang, on both versions and both session servers every week. |
| Tool-call formats             | Whatever SGLang's `--sglang-tool-call-parser` reads for the model; the run used `qwen25` with Qwen3.                                                                                                                                                                                                    |
| Multi-turn tasks              | `end_turn` hands the model each next prompt.                                                                                                                                                                                                                                                            |
| Bundles with several toolsets | Each row lists its own task's tools.                                                                                                                                                                                                                                                                    |
| Envs                          | Wherever the bundle's server deploys them: `build_app(..., sandbox=)` takes any [sandbox provider](https://www.agentenvframework.com/docs/sandboxes.md).                                                                                                                                                                                    |

## Limits

* **A failed rollout leaves its group out.** It can't be played again, since Miles' session holds its
  turns, so its group is left out with it.
* **No LoRA on FSDP.** Miles' FSDP backend trains the whole model.
* **One deploy per rollout.** Every rollout deploys its envs fresh, and the deploy counts in its time.
* **A server killed hard leaves its envs running.** Stop it with SIGTERM, which tears its episodes
  down.
* **Local Docker runs out of networks past about 30 episodes.** With the `local` sandbox, each
  episode of an env that runs in a container gets a Docker network of its own, and Docker's default
  address pools hold about 30. Add wider pools to Docker's `daemon.json`, and restart Docker before
  starting the server:

  ```json title="/etc/docker/daemon.json"
  { "default-address-pools": [{ "base": "10.200.0.0/14", "size": 24 }] }
  ```