Miles

How a rollout runsCopy Anchor Link

Miles, RadixArk's RL framework built from slime, plays agentic rollouts through an agent function. For each sample, it opens a session on its session server, which records every chat request made through it on SGLang, and calls a function you name with the session's URL and the row. AgentEnv's agent function plays the row's task as one episode on a bundle's OpenEnv server, the model chatting through that session. Four pieces connect Miles to a bundle:

Miles' tokenization, SGLang's tool-call parsing and the training stay Miles' own: the session server turns each sample's recorded chat into the tokens Miles trains on, and the task's verifier scores the episode. The helper imports nothing of Miles.

InstallCopy Anchor Link

Miles runs on Linux with NVIDIA GPUs. This page's run used four L40S, training and SGLang sharing them. Run Miles in its image, and install the framework into its Python:

docker run -d --name miles --gpus all --network host --ipc host --shm-size 64g --ulimit memlock=-1 \
  -v "$PWD:$PWD" -w "$PWD" radixark/miles:dev-202611020756 sleep infinity
docker exec miles pip install 'agentenv-framework[miles]>=0.9.1418'

pip check then lists some of the image's own pins as unmet; this page's run trained with them. The bundle's server runs where its envs deploy, with Docker running, in an environment of its own:

pip install 'agentenv-framework[openenv]>=0.9.1418'

Train on GridSimCopy Anchor Link

GridSim is the bundle the TRL page trains on. Each task runs one grid battery for one day in the ERCOT day-ahead market: the model reads the day's 24 hourly prices and submits a schedule, one integer an hour, to charge, sell or idle. The env grades the schedule against the day's exact optimum: 0 for nothing useful, 0.2 for idling all day, 1 for the optimum. The bundle has 2,000 train days and 200 eval days.

Serve the bundleCopy Anchor Link

hf download earakely-scale/GridSimEnv-AgentEnv --repo-type dataset --include "bundles/gridsim-v1/*" --local-dir .

import uvicorn

from agent_env.openenv import build_app

app = build_app("bundles/gridsim-v1", max_concurrent_envs=64)
uvicorn.run(app, host="127.0.0.1", port=8200)

max_concurrent_envs covers a step's 32 rollouts. A rollout that finds the server full waits for room, within reset_timeout.

Write the rowsCopy Anchor Link

python -m agent_env.training.miles http://127.0.0.1:8200 data

wrote all.jsonl, eval.jsonl, train.jsonl to data

The command plays one task of each toolset to list the tools its rows offer, so it deploys an env once. Each row holds the task's system prompt and first prompt as chat messages, and in metadata the split, index and task id the agent function plays, the row's tools, and the server it plays on with its settings. Since the rows carry them, Miles' rollout processes need no settings of their own.

Each chat request carries the row's tools, so Miles' session server renders them with the model's chat template as tools= would anywhere, and SGLang parses the model's calls.

Warm start with SFTCopy Anchor Link

Played as it comes, Qwen3-4B idles: it scores 0.18 on the 200 eval days. Warm-start it with the TRL page's sft.py, in an environment with TRL, so it saves qwen3-4b-gridsim-sft. This page's run trained it in Miles' container on one of the L40S, with trl and peft installed there.

Train with GRPOCopy Anchor Link

This run differs from the other trainers' in one way: Miles' FSDP backend has no LoRA, so it is a full fine-tune, at learning rate 1e-6 constant, across the four GPUs with SGLang colocated. The rest is theirs: the same SFT warm start, 32 rollouts a step (4 days × 8) at temperature 1.1, thinking off, and at most four model turns a rollout. The parts that are about AgentEnv:

python3 /root/miles/train.py \
  --prompt-data data/train.jsonl --eval-prompt-data eval data/eval.jsonl \
  --input-key messages --metadata-key metadata \
  --apply-chat-template-kwargs '{"enable_thinking": false}' \
  --use-session-server --tito-model qwen3 --sglang-tool-call-parser qwen25 \
  --custom-generate-function-path miles.rollout.generate_hub.agentic_tool_call.generate \
  --custom-agent-function-path agent_env.training.miles.agent.run \
  --custom-rm-path agent_env.training.miles.agent.reward \
  --dynamic-sampling-filter-path agent_env.training.miles.agent.keep_scored_groups \
  ...

The whole script, as it ran, is miles_gridsim.sh. Run it in Miles' container from the folder holding qwen3-4b-gridsim-sft and data/. Miles' launcher refuses GPUs it has no verified profile for, as it did the L40S, so the script starts Ray and submits train.py as the launcher does. It evaluates on the 200 eval days before the first step and every 10 steps, one rollout a day at the training temperature.

What it learnsCopy Anchor Link

All start from the same SFT model and score it alike, within what one rollout a day can tell apart (about ±0.05). Through Miles the eval reward rose from 0.54 to 0.65 after 10 steps and 0.70 after 20, most of it on the 4-hour batteries, the hardest days, from 0.33 to 0.60. The share of schedules that scored 0 fell from 12% to 4%. GridSim's greedy rule scores 0.76 on these days.

The train reward, each step's mean over its 4 days, stays near 0.70 from the first step: the 80 days the run drew hold half the eval days' share of 4-hour batteries. Miles' run is a full fine-tune and the others LoRA, and each is one run, so read the gaps as pointers, not a ranking.

A step took about 125 seconds: 61 generating the 32 rollouts and about 50 training. Of the 1,240 episodes played, 3 failed in the env and their groups were left out, and 4 tool calls had arguments that weren't a JSON object, which the model was told.

Settings Miles needsCopy Anchor Link

Envs on another machineCopy Anchor Link

Miles calls the agent function in its rollout processes, and each opens its session on the URL in its row, so the bundle's server must be one those processes reach. Give it a token, which every request and WebSocket but /health must then carry:

import os

import uvicorn

from agent_env.openenv import build_app

app = build_app("bundles/gridsim-v1", max_concurrent_envs=64, token=os.environ["AGENTENV_TOKEN"])
uvicorn.run(app, host="0.0.0.0", port=8200)

python -m agent_env.training.miles http://envs.internal:8200 data --token-var AGENTENV_TOKEN

--token-var names the environment variable that holds the token: the writer reads it, and the rows record its name, never the token. The agent function reads it in each rollout process, so set it in the environment Miles' Ray workers start in. Without a token, anyone who reaches the server deploys and drives its envs.

The server speaks plain HTTP and WebSockets, so the token crosses the network unencrypted. Keep the server on a private network that only Miles' nodes reach, or put a TLS proxy in front of it and give the writer its https:// URL, which the agent function's sessions open as wss://.

Failed rolloutsCopy Anchor Link

A rollout isn't played again when the env fails it (a deploy, a call past its deadline, a server that went away), when the model misses model_timeout, or when its episode ends without a reward: Miles' session already holds the turns the model took. The agent function flags it, and keep_scored_groups leaves its whole group out of the step, and Miles draws more rows until the step's batch is full. Without the filter (--dynamic-sampling-filter-path), a flagged rollout trains as a reward of 0. The filter also leaves out two groups the agent function can't flag:

• Miles aborts a sample whose session recorded no model call, as when the session's first request fails.
• A sample can come back without a reward, as when the agent function's child process died mid-episode. Miles keeps the turns its session recorded but has no metadata to merge.

In an eval, Miles runs no filter, so a flagged rollout scores 0.

Opening its session, a rollout waits for a server at its limit or one restarting, within reset_timeout, so a restart delays rollouts rather than failing them.

A tool call whose arguments aren't a JSON object isn't a failure: the agent function answers it with why, as the model's error, and the episode goes on. The GPU run showed why. Counted as env failures, such calls were left out of training unpenalized, and within six steps the policy drifted into making them.

Miles counts a filter's drops by reason, as rollout/dynamic_filter/drop_<reason>, only when the filter returns Miles' own FilterOutput. keep_scored_groups returns a bool, since the helper imports nothing of Miles, so its drops don't show there. rollout/raw_reward_unfiltered also includes the flagged rollouts' zeros. To count the drops, wrap the filter in one of your own and pass it to --dynamic-sampling-filter-path instead:

from miles.rollout.filter_hub.base_types import FilterOutput

from agent_env.training.miles.agent import keep_scored_groups


def keep_and_count(args, samples, **kwargs):
    if keep_scored_groups(args, samples, **kwargs):
        return FilterOutput(keep=True)
    return FilterOutput(keep=False, reason="agentenv_failed")

With --dynamic-sampling-filter-path my_filters.keep_and_count, Miles logs each step's dropped groups as rollout/dynamic_filter/drop_agentenv_failed.

What the agent function returnsCopy Anchor Link

Miles merges what agent.run returns into each sample's metadata, where the reward hook, the filter and any of your own hooks read it:

Multi-turn tasksCopy Anchor Link

A task with several turns plays as one rollout. Each row offers end_turn with the task's tools: the model calls it with its answer, and its result is the next turn's prompt, which the agent function returns like any tool answer. When the model replies without calling a tool, its reply ends the turn it left open, and any turns after it end empty. Tell the model in the system prompt to end each turn with end_turn. A task that records several scores needs a reward that combines them, given to build_app, here their mean:

app = build_app("bundles/my-bundle", reward=lambda scores: sum(scores.values()) / len(scores),
                max_concurrent_envs=64)

What's supportedCopy Anchor Link

LimitsCopy Anchor Link

• A failed rollout leaves its group out. It can't be played again, since Miles' session holds its turns, so its group is left out with it.

• No LoRA on FSDP. Miles' FSDP backend trains the whole model.

• One deploy per rollout. Every rollout deploys its envs fresh, and the deploy counts in its time.

• A server killed hard leaves its envs running. Stop it with SIGTERM, which tears its episodes down.

• Local Docker runs out of networks past about 30 episodes. With the local sandbox, each episode of an env that runs in a container gets a Docker network of its own, and Docker's default address pools hold about 30. Add wider pools to Docker's daemon.json, and restart Docker before starting the server:

  Copy Text

  { "default-address-pools": [{ "base": "10.200.0.0/14", "size": 24 }] }