slime

How a rollout runsCopy Anchor Link

slime, THUDM's RL post-training framework on Megatron and SGLang, generates each rollout with a function you name, and its agent layer, slime.agent, gives that function an OpenAI chat endpoint on SGLang that records every turn of a session as training samples. AgentEnv's generate function opens a session for each sample, plays the row's task as one episode on a bundle's OpenEnv server, the model chatting through that session, and hands slime the session's turns with the episode's reward. Four pieces connect slime to a bundle:

slime's tokenization, SGLang's generation and the training stay slime's own, and it calls no reward model: the task's verifier scores each episode, and the adapter sets that score on every sample the session's turns become.

InstallCopy Anchor Link

slime runs on Linux with NVIDIA GPUs, and trains on Megatron. This page's runs used four L40S, training and SGLang sharing them. Run slime in its image, and install the framework into its Python:

docker run -d --name slime --gpus all --network host --ipc host --shm-size 64g --ulimit memlock=-1 \
  -v "$PWD:/work" slimerl/slime:latest sleep infinity
docker exec slime pip install 'agentenv-framework[slime]>=0.9.1421'

The helper needs slime main from 540ea91 (October 27, 2026), the commit this page's runs used: the adapter in slime's latest release, v0.4.0, predates the settings it passes. The bundle's server runs where its envs deploy, with Docker running, in an environment of its own:

pip install 'agentenv-framework[openenv]>=0.9.1421'

Train on GridSimCopy Anchor Link

GridSim is the bundle the TRL page trains on. Each task runs one grid battery for one day in the ERCOT day-ahead market: the model reads the day's 24 hourly prices and submits a schedule, one integer an hour, to charge, sell or idle. The env grades the schedule against the day's exact optimum: 0 for nothing useful, 0.2 for idling all day, 1 for the optimum. The bundle has 2,000 train days and 200 eval days.

Serve the bundleCopy Anchor Link

hf download earakely-scale/GridSimEnv-AgentEnv --repo-type dataset --include "bundles/gridsim-v1/*" --local-dir .

import uvicorn

from agent_env.openenv import build_app

app = build_app("bundles/gridsim-v1", max_concurrent_envs=64)
uvicorn.run(app, host="127.0.0.1", port=8200)

max_concurrent_envs covers a step's 32 rollouts. A rollout that finds the server full waits for room, within reset_timeout.

Write the rowsCopy Anchor Link

python -m agent_env.training.slime http://127.0.0.1:8200 data

wrote all.jsonl, eval.jsonl, train.jsonl to data

The command plays one task of each toolset to list the tools its rows offer, so it deploys an env once. Each row holds the task's system prompt and first prompt as chat messages, and in metadata the split, index and task id the generate function plays, the row's tools, and the server it plays on with its settings. They are the rows the Miles helper writes, with the same options: --max-turns (4), --tool-timeout (60 s), --reset-timeout (600 s), --model-timeout (600 s) and --token-var.

Warm start with SFTCopy Anchor Link

Played as it comes, Qwen3-4B idles: it scores 0.18 on the 200 eval days. Warm-start it with the TRL page's sft.py, in an environment with TRL, so it saves qwen3-4b-gridsim-sft. This page's runs trained it in slime's container on one of the L40S, with trl and peft installed there. slime's Megatron trainer loads a torch_dist checkpoint, so convert it in slime's container:

cd /root/slime && source scripts/models/qwen3-4B.sh
PYTHONPATH=/root/Megatron-LM torchrun --nproc-per-node 1 tools/convert_hf_to_torch_dist.py "${MODEL_ARGS[@]}" \
  --hf-checkpoint /work/qwen3-4b-gridsim-sft --save /work/qwen3-4b-gridsim-sft_torch_dist

Train with GRPOCopy Anchor Link

slime trains on Megatron with no LoRA, so this is a full fine-tune, at learning rate 1e-6 constant, across the four GPUs with SGLang colocated, as the Miles run was. The rest is the other trainers' runs' settings: the same SFT warm start, 32 rollouts a step (4 days × 8) at temperature 1.1, thinking off, and at most four model turns a rollout. The parts that are about AgentEnv:

python3 train.py \
  --prompt-data /work/data/train.jsonl --eval-prompt-data eval /work/data/eval.jsonl \
  --input-key messages --metadata-key metadata \
  --apply-chat-template-kwargs '{"enable_thinking": false}' \
  --sglang-tool-call-parser qwen25 \
  --custom-generate-function-path agent_env.training.slime.generate.generate \
  --dynamic-sampling-filter-path agent_env.training.slime.generate.keep_scored_groups \
  ...

The whole script, as it ran, is slime_gridsim.sh. Run it in slime's container from /root/slime; it starts Ray and submits train.py, as slime's own scripts do. It evaluates on the 200 eval days before the first step and every 20 steps, one rollout a day at the training temperature.

What it learnsCopy Anchor Link

Through slime the eval reward rose from 0.51 to 0.62 after 20 steps and 0.69 after 40 (standard errors about 0.025), on every battery size and most on the 4-hour batteries, the hardest days, from 0.30 to 0.61. The share of schedules that scored 0 fell from 21% to 3%. GridSim's greedy rule scores 0.76 on these days. The train reward, each step's mean over its 4 days, stays near 0.68 throughout.

An earlier 20-step run from another SFT warm start went from 0.58 to 0.60, within what one rollout a day can tell apart; as with SkyRL and TRL, 20 steps weren't enough to show it. slime's run and Miles' are full fine-tunes and the others LoRA, and each is one run, so read the gaps as pointers, not a ranking.

A step took about 107 seconds: 31 generating the 32 rollouts and 58 training. No rollout failed and no tool call was malformed in the 1,880 episodes.

Settings slime needsCopy Anchor Link

On L40S the run also needed --attention-backend unfused, since slime's image has no FlashAttention-2: flash compiles FlashAttention 4's kernels, which the L40S can't run, and fused doesn't take packed sequences there. On 48 GB GPUs it needed --max-tokens-per-gpu 4096, without --accumulate-allreduce-grads-in-fp32, to fit the backward.

A sample per turnCopy Anchor Link

The adapter joins each turn of a session to the one before into one training sample when the turn's prompt extends the tokens it already holds. Qwen3's chat template, with thinking off, puts an empty think block in the prompt the model answers but leaves it out when it renders that answer as history, so each turn of a GridSim episode came back as a sample of its own, about 2.5 an episode. Each sample holds the context the model saw and trains only its own turn, with the episode's reward. slime computes each group's baseline over rollouts, not samples, and gives every sample of a rollout its advantage, so an episode with more turns doesn't weigh more in training.

In an eval, the generate function returns one sample per episode, carrying only its reward, so slime's eval/eval is the mean over episodes. This page's runs predate that: their eval/eval averaged over turns, 0.50, 0.58 and 0.62 at steps 0, 20 and 40, where the means per episode, as the table above gives them, are 0.51, 0.62 and 0.69.

Envs on another machineCopy Anchor Link

slime calls the generate function in its rollout process, and each session opens on the URL in its row, so the bundle's server must be one that process reaches. Give it a token, which every request and WebSocket but /health must then carry:

import os

import uvicorn

from agent_env.openenv import build_app

app = build_app("bundles/gridsim-v1", max_concurrent_envs=64, token=os.environ["AGENTENV_TOKEN"])
uvicorn.run(app, host="0.0.0.0", port=8200)

python -m agent_env.training.slime http://envs.internal:8200 data --token-var AGENTENV_TOKEN

--token-var names the environment variable that holds the token: the writer reads it, and the rows record its name, never the token. The generate function reads it in slime's rollout process, so set it in the environment slime's Ray workers start in. Without a token, anyone who reaches the server deploys and drives its envs.

The server speaks plain HTTP and WebSockets, so the token crosses the network unencrypted. Keep the server on a private network that only slime's nodes reach, or put a TLS proxy in front of it and give the writer its https:// URL, which the sessions open as wss://.

Failed rolloutsCopy Anchor Link

A rollout isn't played again when the env fails it (a deploy, a call past its deadline, a server that went away), when the model misses model_timeout, or when its episode ends without a reward: the adapter already holds the turns the model took. The generate function flags its samples, and keep_scored_groups leaves its whole group out of the step, and slime draws more rows until the step's batch is full. A rollout whose session recorded no turn, as when the model's first request fails, comes back as one aborted sample with remove_sample, and its group is left out too. In an eval, slime runs no filter, so a flagged rollout scores 0.

When a step's batch is full, slime cancels the rollouts still playing, which the generate function takes (abort_mode = "request"): each drops its session and closes its episode rather than playing on against the aborted model.

Opening its session, a rollout waits for a server at its limit or one restarting, within reset_timeout, so a restart delays rollouts rather than failing them.

A tool call whose arguments aren't a JSON object isn't a failure: the generate function answers it with why, as the model's error, and the episode goes on.

What the samples carryCopy Anchor Link

The adapter merges the episode's metadata into every sample it returns, where the filter and any of your own hooks read it:

Multi-turn tasksCopy Anchor Link

A task with several turns plays as one rollout. Each row offers end_turn with the task's tools: the model calls it with its answer, and its result is the next turn's prompt, which the generate function returns like any tool answer. When the model replies without calling a tool, its reply ends the turn it left open, and any turns after it end empty. Tell the model in the system prompt to end each turn with end_turn. A task that records several scores needs a reward that combines them, given to build_app, here their mean:

app = build_app("bundles/my-bundle", reward=lambda scores: sum(scores.values()) / len(scores),
                max_concurrent_envs=64)

What's supportedCopy Anchor Link

LimitsCopy Anchor Link

• A failed rollout leaves its group out. It can't be played again, since the adapter holds its turns, so its group is left out with it.

• No LoRA. slime trains the whole model on Megatron.

• One deploy per rollout. Every rollout deploys its envs fresh, and the deploy counts in its time.

• A server killed hard leaves its envs running. Stop it with SIGTERM, which tears its episodes down.

• Local Docker runs out of networks past about 30 episodes. With the local sandbox, each episode of an env that runs in a container gets a Docker network of its own, and Docker's default address pools hold about 30. Add wider pools to Docker's daemon.json, and restart Docker before starting the server:

  Copy Text

  { "default-address-pools": [{ "base": "10.200.0.0/14", "size": 24 }] }