# slime (https://www.agentenvframework.com/docs/training/slime)

> Train using slime

## How a rollout runs

1. slime’s rollout calls the helper’s `generate` for a sample of the split’s JSON Lines, which opens a session for it on slime’s OpenAI adapter and starts the row’s task on the bundle’s OpenEnv server.
2. The server starts the task as an episode: it deploys the env, loads the task, and opens the agent’s turn, while SGLang writes the model’s first reply through the adapter, which records it.
3. The adapter reads the reply’s tool calls with `--sglang-tool-call-parser`; each runs on the episode, and its answer goes back as a tool message, turn after turn up to the row’s `max_turns`.
4. A reply without a tool call ends the turn it left open; a task with another turn answers `end_turn` with the next prompt. The task’s verifier scores the episode.
5. `finish_session` turns the recorded turns into samples carrying the score, so slime calls no reward model; a rollout the env failed leaves its group out. The episode is torn down.

Parts of the scene:

- **slime**: slime’s trainer generates each rollout with the function `--custom-generate-function-path` names, on its own SGLang, and trains on the samples it returns, on Megatron. slime owns the model, the generation and the update; the environment owns the reward.
- **A dataset row**: A line of `<split>.jsonl`: the task’s prompt as chat `messages`, and in `metadata` the `split`, `index` and `task_id` the generate function plays, the row’s tools, and the server’s settings. The prompt is yours to change; the task stays the same.
- **generate**: slime’s generate function, called once per sample on its rollout’s event loop: it opens a session on slime’s OpenAI adapter, posts the conversation and the row’s tools to it, calls each tool call on the episode, and answers a call whose arguments aren’t a JSON object with why.
- **The session**: One OpenEnv WebSocket session per rollout, waiting out a server at its limit or restarting, within `reset_timeout`. A rollout the env fails isn’t played again, since the adapter already holds the model’s turns.
- **The OpenEnv server**: `agent_env.openenv.build_app(bundle)` serves every task of a bundle as an OpenEnv env, the split `all` plus one per eval. With `token=`, it can run on another machine than slime’s.
- **The env**: The task’s env, deployed fresh for this episode in a sandbox, here a container on your Docker. Its tools are what the model calls.
- **The agent’s turn**: The task prompts one agent, and your model plays it from outside: slime’s OpenAI adapter renders each request with the model’s chat template and the row’s tools, and the tool calls it reads go to the episode.
- **The verifier**: The task’s own verifier step scores the episode once the turn ends, the same score an evaluation run records. With several scores, `reward=` picks the one to train on.
- **The reward**: The adapter’s `finish_session` sets the episode’s reward on every sample the session’s turns become, so slime runs no reward model. An episode the env failed, or that ended without a reward, is flagged, and `keep_scored_groups`, slime’s dynamic-sampling filter, leaves its group out of the step.

[slime](https://github.com/THUDM/slime), THUDM's RL post-training framework on Megatron and SGLang,
generates each rollout with a function you name, and its agent layer, `slime.agent`, gives that function an OpenAI
chat endpoint on SGLang that records every turn of a session as training samples. AgentEnv's generate
function opens a session for each sample, plays the row's task as one episode on a bundle's OpenEnv
server, the model chatting through that session, and hands slime the session's turns with the
episode's reward. Four pieces connect slime to a bundle:

| Piece                                                  | What it does                                                                                                                                                                                                        |
| ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `agent_env.openenv.build_app(bundle)`                  | Serves every task of a bundle as an [OpenEnv](https://www.agentenvframework.com/docs/openenv.md) env: the split `all`, plus one split per eval.                                                                                                         |
| `python -m agent_env.training.slime`                   | Writes the bundle's rows as slime's prompt data, a JSON Lines file per split: each row's chat messages, and in its metadata the task, its tools and the server's settings.                                          |
| `agent_env.training.slime.generate.generate`           | The generate function (`--custom-generate-function-path`): opens a session on slime's OpenAI adapter, plays the row's episode through it, and returns the session's turns as samples carrying the episode's reward. |
| `agent_env.training.slime.generate.keep_scored_groups` | The dynamic-sampling filter (`--dynamic-sampling-filter-path`), which leaves out the group of an episode the env failed.                                                                                            |

slime's tokenization, SGLang's generation and the training stay slime's own, and it calls no reward
model: the task's verifier scores each episode, and the adapter sets that score on every sample the
session's turns become.

## Install

slime runs on Linux with NVIDIA GPUs, and trains on Megatron. This page's runs used four L40S,
training and SGLang sharing them. Run slime in its image, and install the framework into its Python:

```bash title="Terminal"
docker run -d --name slime --gpus all --network host --ipc host --shm-size 64g --ulimit memlock=-1 \
  -v "$PWD:/work" slimerl/slime:latest sleep infinity
docker exec slime pip install 'agentenv-framework[slime]>=0.9.1421'
```

The helper needs slime main from 540ea91 (October 27, 2026), the commit this page's runs used: the
adapter in slime's latest release, v0.4.0, predates the settings it passes. The bundle's server runs
where its envs deploy, with Docker running, in an environment of its own:

```bash title="Terminal"
pip install 'agentenv-framework[openenv]>=0.9.1421'
```

## Train on GridSim

GridSim is the bundle the [TRL page](https://www.agentenvframework.com/docs/training/trl.md#train-on-gridsim) trains on. Each task runs
one grid battery for one day in the ERCOT day-ahead market: the model reads the day's 24 hourly
prices and submits a schedule, one integer an hour, to charge, sell or idle. The env grades the
schedule against the day's exact optimum: 0 for nothing useful, 0.2 for idling all day, 1 for the
optimum. The bundle has 2,000 train days and 200 eval days.

### Serve the bundle

```bash title="Terminal"
hf download earakely-scale/GridSimEnv-AgentEnv --repo-type dataset --include "bundles/gridsim-v1/*" --local-dir .
```

```python title="serve.py"
import uvicorn

from agent_env.openenv import build_app

app = build_app("bundles/gridsim-v1", max_concurrent_envs=64)
uvicorn.run(app, host="127.0.0.1", port=8200)
```

`max_concurrent_envs` covers a step's 32 rollouts. A rollout that finds the server full waits for
room, within `reset_timeout`.

### Write the rows

```bash title="Terminal"
python -m agent_env.training.slime http://127.0.0.1:8200 data
```

```text title="Output"
wrote all.jsonl, eval.jsonl, train.jsonl to data
```

The command plays one task of each toolset to list the tools its rows offer, so it deploys an env
once. Each row holds the task's system prompt and first prompt as chat `messages`, and in `metadata`
the split, index and task id the generate function plays, the row's tools, and the server it plays on
with its settings. They are the rows the [Miles helper](https://www.agentenvframework.com/docs/training/miles.md#write-the-rows) writes,
with the same options: `--max-turns` (4), `--tool-timeout` (60 s), `--reset-timeout` (600 s),
`--model-timeout` (600 s) and `--token-var`.

### Warm start with SFT

Played as it comes, Qwen3-4B idles: it scores 0.18 on the 200 eval days. Warm-start it with the
TRL page's [`sft.py`](https://www.agentenvframework.com/docs/training/trl.md#warm-start-with-sft), in an environment with TRL, so it saves
`qwen3-4b-gridsim-sft`. This page's runs trained it in slime's container on one of the L40S, with
`trl` and `peft` installed there. slime's Megatron trainer loads a torch\_dist checkpoint, so convert
it in slime's container:

```bash title="Terminal"
cd /root/slime && source scripts/models/qwen3-4B.sh
PYTHONPATH=/root/Megatron-LM torchrun --nproc-per-node 1 tools/convert_hf_to_torch_dist.py "${MODEL_ARGS[@]}" \
  --hf-checkpoint /work/qwen3-4b-gridsim-sft --save /work/qwen3-4b-gridsim-sft_torch_dist
```

### Train with GRPO

slime trains on Megatron with no LoRA, so this is a full fine-tune, at learning rate 1e-6 constant,
across the four GPUs with SGLang colocated, as the [Miles run](https://www.agentenvframework.com/docs/training/miles.md#train-with-grpo)
was. The rest is the other trainers' runs' settings: the same SFT warm start, 32 rollouts a step (4
days × 8) at temperature 1.1, thinking off, and at most four model turns a rollout. The parts that
are about AgentEnv:

```bash title="slime_gridsim.sh (excerpt)"
python3 train.py \
  --prompt-data /work/data/train.jsonl --eval-prompt-data eval /work/data/eval.jsonl \
  --input-key messages --metadata-key metadata \
  --apply-chat-template-kwargs '{"enable_thinking": false}' \
  --sglang-tool-call-parser qwen25 \
  --custom-generate-function-path agent_env.training.slime.generate.generate \
  --dynamic-sampling-filter-path agent_env.training.slime.generate.keep_scored_groups \
  ...
```

The whole script, as it ran, is [`slime_gridsim.sh`](/training/slime_gridsim.sh). Run it in slime's
container from `/root/slime`; it starts Ray and submits `train.py`, as slime's own scripts do. It
evaluates on the 200 eval days before the first step and every 20 steps, one rollout a day at the
training temperature.

### What it learns

| Qwen3-4B | eval reward (200 days) |
|---|---|
| after SFT (GRPO step 0) | 0.514 |
| GRPO step 20 | 0.615 |
| GRPO step 40 | 0.694 |

Mean train reward of each GRPO step's 32 rollouts, by step: 1: 0.749, 2: 0.777, 3: 0.720, 4: 0.687, 5: 0.672, 6: 0.624, 7: 0.599, 8: 0.757, 9: 0.679, 10: 0.549, 11: 0.465, 12: 0.464, 13: 0.662, 14: 0.847, 15: 0.641, 16: 0.593, 17: 0.915, 18: 0.725, 19: 0.786, 20: 0.693, 21: 0.592, 22: 0.578, 23: 0.560, 24: 0.710, 25: 0.583, 26: 0.828, 27: 0.577, 28: 0.697, 29: 0.496, 30: 0.756, 31: 0.681, 32: 0.808, 33: 0.717, 34: 0.806, 35: 0.776, 36: 0.616, 37: 0.880, 38: 0.700, 39: 0.621, 40: 0.679.

| Qwen3-4B, eval reward on the 200 eval days | slime | Miles   | SkyRL | verl | prime-rl | NeMo RL | TRL  | Unsloth |
| ------------------------------------------ | ----- | ------- | ----- | ---- | -------- | ------- | ---- | ------- |
| After SFT (GRPO step 0)                    | 0.51  | 0.54    | 0.54  | 0.52 | 0.56     | 0.55    | 0.57 | 0.55    |
| After 20 GRPO steps                        | 0.62  | 0.70    | 0.55  | 0.59 | 0.58     | 0.68    | 0.56 | 0.52    |
| After 40 GRPO steps                        | 0.69  | not run | 0.61  | 0.60 | 0.57     | 0.67    | 0.61 | 0.59    |

Through slime the eval reward rose from 0.51 to 0.62 after 20 steps and 0.69 after 40 (standard
errors about 0.025), on every battery size and most on the 4-hour batteries, the hardest days, from
0.30 to 0.61. The share of schedules that scored 0 fell from 21% to 3%. GridSim's greedy rule scores
0.76 on these days. The train reward, each step's mean over its 4 days, stays near 0.68 throughout.

An earlier 20-step run from another SFT warm start went from 0.58 to 0.60, within what one rollout a
day can tell apart; as with SkyRL and TRL, 20 steps weren't enough to show it. slime's run and Miles'
are full fine-tunes and the others LoRA, and each is one run, so read the gaps as pointers, not a
ranking.

A step took about 107 seconds: 31 generating the 32 rollouts and 58 training. No rollout failed and
no tool call was malformed in the 1,880 episodes.

## Settings slime needs

| Setting                         | Value here                   | What it does                                                                                                                                                                                                       |
| ------------------------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--sglang-tool-call-parser`     | `qwen25`                     | The parser slime's adapter reads the model's tool calls with, for its chat template. The generate function refuses to start without one: with none, the model's calls stay in its text and every rollout scores 0. |
| `--input-key`, `--metadata-key` | `messages`, `metadata`       | Where the rows hold the chat and what the generate function plays.                                                                                                                                                 |
| `--apply-chat-template-kwargs`  | `{"enable_thinking": false}` | The template settings the adapter renders each session with, here as the SFT data was.                                                                                                                             |
| `--rollout-max-context-len`     | 8192                         | The context a session may grow to.                                                                                                                                                                                 |
| `--global-batch-size`           | 32                           | Rollouts a training step takes, not samples: a rollout's samples stay in one step.                                                                                                                                 |

On L40S the run also needed `--attention-backend unfused`, since slime's image has no
FlashAttention-2: `flash` compiles FlashAttention 4's kernels, which the L40S can't run, and `fused`
doesn't take packed sequences there. On 48 GB GPUs it needed
`--max-tokens-per-gpu 4096`, without `--accumulate-allreduce-grads-in-fp32`, to fit the backward.

## A sample per turn

The adapter joins each turn of a session to the one before into one training sample when the turn's
prompt extends the tokens it already holds. Qwen3's chat template, with thinking off, puts an empty
think block in the prompt the model answers but leaves it out when it renders that answer as history,
so each turn of a GridSim episode came back as a sample of its own, about 2.5 an episode. Each sample
holds the context the model saw and trains only its own turn, with the episode's reward. slime
computes each group's baseline over rollouts, not samples, and gives every sample of a rollout its
advantage, so an episode with more turns doesn't weigh more in training.

In an eval, the generate function returns one sample per episode, carrying only its reward, so slime's
`eval/eval` is the mean over episodes. This page's runs predate that: their `eval/eval` averaged over
turns, 0.50, 0.58 and 0.62 at steps 0, 20 and 40, where the means per episode, as the table above
gives them, are 0.51, 0.62 and 0.69.

## Envs on another machine

slime calls the generate function in its rollout process, and each session opens on the URL in its
row, so the bundle's server must be one that process reaches. Give it a token, which every request
and WebSocket but `/health` must then carry:

```python title="serve.py"
import os

import uvicorn

from agent_env.openenv import build_app

app = build_app("bundles/gridsim-v1", max_concurrent_envs=64, token=os.environ["AGENTENV_TOKEN"])
uvicorn.run(app, host="0.0.0.0", port=8200)
```

```bash title="Terminal"
python -m agent_env.training.slime http://envs.internal:8200 data --token-var AGENTENV_TOKEN
```

`--token-var` names the environment variable that holds the token: the writer reads it, and the rows
record its name, never the token. The generate function reads it in slime's rollout process, so set
it in the environment slime's Ray workers start in. Without a token, anyone who reaches the server
deploys and drives its envs.

The server speaks plain HTTP and WebSockets, so the token crosses the network unencrypted. Keep the
server on a private network that only slime's nodes reach, or put a TLS proxy in front of it and give
the writer its `https://` URL, which the sessions open as `wss://`.

## Failed rollouts

A rollout isn't played again when the env fails it (a deploy, a call past its deadline, a server
that went away), when the model misses `model_timeout`, or when its episode ends without a reward:
the adapter already holds the turns the model took. The generate function flags its samples, and
`keep_scored_groups` leaves its whole group out of the step, and slime draws more rows until the
step's batch is full. A rollout whose session recorded no turn, as when the model's first request
fails, comes back as one aborted sample with `remove_sample`, and its group is left out too. In an
eval, slime runs no filter, so a flagged rollout scores 0.

When a step's batch is full, slime cancels the rollouts still playing, which the generate function
takes (`abort_mode = "request"`): each drops its session and closes its episode rather than playing on
against the aborted model.

Opening its session, a rollout waits for a server at its limit or one restarting, within
`reset_timeout`, so a restart delays rollouts rather than failing them.

A tool call whose arguments aren't a JSON object isn't a failure: the generate function answers it
with why, as the model's error, and the episode goes on.

## What the samples carry

The adapter merges the episode's metadata into every sample it returns, where the filter and any of
your own hooks read it:

| Key                      | What it holds                                                                                               |
| ------------------------ | ----------------------------------------------------------------------------------------------------------- |
| `reward`                 | The episode's reward, or 0 for a flagged rollout; also each sample's `reward`.                              |
| `agentenv_scores`        | Each score the task's verifiers recorded, by the id of the step that recorded it. The reward combines them. |
| `agentenv_tool_calls`    | The tool calls run on the episode.                                                                          |
| `agentenv_invalid_calls` | The tool calls whose arguments weren't a JSON object, answered with why.                                    |
| `agentenv_failure`       | Why the rollout was flagged, when it was.                                                                   |

## Multi-turn tasks

A task with several turns plays as one rollout. Each row offers `end_turn` with the task's tools: the
model calls it with its answer, and its result is the next turn's prompt, which the generate function
returns like any tool answer. When the model replies without calling a tool, its reply ends the turn
it left open, and any turns after it end empty. Tell the model in the system prompt to end each turn
with `end_turn`. A task that records several scores needs a reward that combines them, given to
`build_app`, here their mean:

```python title="serve.py"
app = build_app("bundles/my-bundle", reward=lambda scores: sum(scores.values()) / len(scores),
                max_concurrent_envs=64)
```

## What's supported

|                               |                                                                                                                                                                                                                                                                                                     |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| slime                         | main from 540ea91. The page's runs used `slimerl/slime:latest` at 540ea91; a CI job plays rows through the generate function on slime's own adapter, with slime's own arguments and a stand-in SGLang, on 540ea91 and on main every week. v0.4.0's adapter predates the settings the helper passes. |
| Tool-call formats             | Whatever slime's adapter reads with `--sglang-tool-call-parser`; the runs used `qwen25` with Qwen3.                                                                                                                                                                                                 |
| Multi-turn tasks              | `end_turn` hands the model each next prompt.                                                                                                                                                                                                                                                        |
| Bundles with several toolsets | Each row lists its own task's tools.                                                                                                                                                                                                                                                                |
| Envs                          | Wherever the bundle's server deploys them: `build_app(..., sandbox=)` takes any [sandbox provider](https://www.agentenvframework.com/docs/sandboxes.md).                                                                                                                                                                                |

## Limits

* **A failed rollout leaves its group out.** It can't be played again, since the adapter holds its
  turns, so its group is left out with it.
* **No LoRA.** slime trains the whole model on Megatron.
* **One deploy per rollout.** Every rollout deploys its envs fresh, and the deploy counts in its time.
* **A server killed hard leaves its envs running.** Stop it with SIGTERM, which tears its episodes
  down.
* **Local Docker runs out of networks past about 30 episodes.** With the `local` sandbox, each
  episode of an env that runs in a container gets a Docker network of its own, and Docker's default
  address pools hold about 30. Add wider pools to Docker's `daemon.json`, and restart Docker before
  starting the server:

  ```json title="/etc/docker/daemon.json"
  { "default-address-pools": [{ "base": "10.200.0.0/14", "size": 24 }] }
  ```