verl

How a rollout runsCopy Anchor Link

verl plays multi-turn rollouts in agent loops: for each row, an agent loop generates on verl's own vLLM, parses the model's tool calls, adds their answers to the context, and returns the tokens and the reward. AgentEnv's agent loop plays each row's task as one episode on a bundle's OpenEnv server. Three pieces connect verl to a bundle:

verl's tokenization, tool parsers, turn caps and truncation stay verl's own, and verl calls no reward function: the task's verifier scores each episode.

InstallCopy Anchor Link

verl runs on Linux with NVIDIA GPUs. This page's runs used one L40S, the trainer and vLLM sharing it. Run verl in its CUDA 13 image, whose uv cache holds every wheel its lockfile names, and install the framework into its environment:

docker run -d --name verl --gpus all --network host --ipc host --shm-size 32g \
  -v "$PWD:$PWD" -w "$PWD" verlai/verl:uv.cu130.dev5 sleep infinity
docker exec verl bash -c "cd /workspace/verl && uv sync --frozen --offline --extra vllm --extra fsdp"
docker exec verl uv pip install --python /workspace/verl/.venv/bin/python 'agentenv-framework[verl]>=0.9.1401'

Run verl's commands with docker exec verl /workspace/verl/.venv/bin/python .... The bundle's server runs where its envs deploy, with Docker running, in an environment of its own:

pip install 'agentenv-framework[openenv]>=0.9.1401'

Train on GridSimCopy Anchor Link

GridSim is the bundle the TRL page trains on. Each task runs one grid battery for one day in the ERCOT day-ahead market: the model reads the day's 24 hourly prices and submits a schedule, one integer an hour, to charge, sell or idle. The env grades the schedule against the day's exact optimum: 0 for nothing useful, 0.2 for idling all day, 1 for the optimum. The bundle has 2,000 train days and 200 eval days.

Serve the bundleCopy Anchor Link

hf download earakely-scale/GridSimEnv-AgentEnv --repo-type dataset --include "bundles/gridsim-v1/*" --local-dir .

import uvicorn

from agent_env.openenv import build_app

app = build_app("bundles/gridsim-v1", max_concurrent_envs=64)
uvicorn.run(app, host="127.0.0.1", port=8200)

max_concurrent_envs covers a step's 32 rollouts, or the two batches the async trainer keeps going; an eval's 200 rollouts wait their turn for a session.

Write the rowsCopy Anchor Link

python -m agent_env.training.verl http://127.0.0.1:8200 data \
  --tokenizer Qwen/Qwen3-4B --max-prompt-length 3072 --chat-template-kwargs '{"enable_thinking": false}'

wrote agent_loop.yaml, all.parquet, eval.parquet, train.parquet to data
the longest prompt is 1169 tokens with its tools

The command plays one task of each toolset to list the tools its rows offer, so it deploys an env once. Each row holds the task's prompt as chat messages, agent_name: agentenv, and in extra_info the split, index and task id the agent loop plays, with the row's tools.

--tokenizer counts each row's prompt with its tools as the model's chat template renders them, with verl's data.apply_chat_template_kwargs given as --chat-template-kwargs. A row over --max-prompt-length is refused before anything is written: verl would cut it from the left, losing the system prompt and the tools, and verl's own length filter doesn't count a row's tools.

- name: agentenv
  _target_: agent_env.training.verl.loop.AgentEnvAgentLoop
  url: http://127.0.0.1:8200
  tool_timeout: 60.0
  reset_timeout: 600.0
  attempts: 3

Warm start with SFTCopy Anchor Link

Played as it comes, Qwen3-4B idles: it scores 0.18 on the 200 eval days. Warm-start it with the TRL page's sft.py, in an environment with TRL, so it saves qwen3-4b-gridsim-sft. It took 25 minutes on one L40S.

Train with GRPOCopy Anchor Link

This run uses the TRL, NeMo RL and prime-rl runs' settings: the same SFT warm start, LoRA r32/α64 on every linear layer, learning rate 2e-5 cosine with 3 warmup steps, 32 rollouts a step (4 days × 8) at temperature 1.1, thinking off, and at most four model turns a rollout. The parts that are about AgentEnv:

python -m verl.trainer.main_ppo \
  data.train_files=data/train.parquet data.val_files=data/eval.parquet \
  data.return_raw_chat=True +data.apply_chat_template_kwargs.enable_thinking=False \
  actor_rollout_ref.rollout.multi_turn.enable=True \
  actor_rollout_ref.rollout.multi_turn.format=hermes \
  actor_rollout_ref.rollout.multi_turn.max_assistant_turns=4 \
  actor_rollout_ref.rollout.multi_turn.max_tool_response_length=8192 \
  actor_rollout_ref.rollout.agent.agent_loop_config_path=data/agent_loop.yaml \
  actor_rollout_ref.rollout.agent.default_agent_loop=agentenv \
  actor_rollout_ref.rollout.layered_summon=False \
  trainer.v1.sampler.sync_refill_failed_groups=True \
  ...

The whole command, as it ran, is verl_gridsim.sh. Run it in verl's environment from the folder holding qwen3-4b-gridsim-sft and data/. It evaluates on the 200 eval days before the first step and every 20 steps, one rollout a day at the training temperature.

step:21 - critic/score/mean:0.6845 - response_length/mean:815.75 - num_turns/mean:7.94 - timing_s/gen:32.77 - timing_s/step:63.01 - …

critic/score/mean is the mean of the step's 32 rewards. verl counts the user and tool turns too, so eight turns are the four model turns the cap allows.

What it learnsCopy Anchor Link

All four start from the same SFT model and score it alike, within what one rollout a day can tell apart (about ±0.06). Through verl the train reward rises from 0.59 over the first 10 steps to 0.69 over the last 20, and the eval reward from 0.52 to 0.60; a second run, with the envs on the GPU machine, scored 0.52, 0.59 and 0.60. GridSim's greedy rule scores 0.76 on these days. This is one run of each trainer, so read the gaps as pointers, not a ranking.

The 40 GRPO steps with the three evals took 67 minutes on one L40S: about 75 seconds a step, a little over half of it generation.

Settings verl needsCopy Anchor Link

Envs on another machineCopy Anchor Link

verl's agent-loop workers can run on any node of its Ray cluster, and each opens its sessions on the URL in agent_loop.yaml, so the bundle's server must be one every node reaches. Give it a token, which every request and WebSocket but /health must then carry:

app = build_app("bundles/gridsim-v1", max_concurrent_envs=64, token=os.environ["AGENTENV_TOKEN"])
uvicorn.run(app, host="0.0.0.0", port=8200)

python -m agent_env.training.verl http://envs.internal:8200 data --token-var AGENTENV_TOKEN ...

--token-var names the environment variable that holds the token: the writer reads it, and agent_loop.yaml records its name, never the token, for the agent loop to read in each worker. Set it where verl starts, which Ray workers inherit, or in ray_kwargs.ray_init.runtime_env.env_vars. Without a token, anyone who reaches the server deploys and drives its envs.

The server speaks plain HTTP and WebSockets, so the token crosses the network unencrypted. Keep the server on a private network that only verl's nodes reach, as the run plotted above did, or put a TLS proxy in front of it and give the writer its https:// URL, which the agent loop's sessions open as wss://.

The run plotted above served GridSim this way, from a CPU machine of its own in the same network as the GPU one, and scored 0.52, 0.59 and 0.60 at steps 0, 20 and 40.

Failed rolloutsCopy Anchor Link

A rollout the env fails, a deploy, a call past its deadline or a server that went away, is played again from the start on a new episode, a fresh sample of the same policy: up to attempts times, but none starts once reset_timeout has passed since the rollout began. One the env fails on every attempt raises RolloutFailed, and verl's V1 trainer, the default, records the session as failed and leaves the rollout out of the loss, its group's baseline and the eval reward. A step left with a group of no rollouts at all draws a fresh row for it with sync_refill_failed_groups.

Opening its session, a rollout waits for a server at its limit or one restarting, within reset_timeout, so a restart delays rollouts rather than failing them. In a test run the env server was restarted twice while rollouts opened their sessions; those steps ran up to 54 seconds longer and lost no rollout. In another, with one attempt a rollout and the server stopped mid-episode, the rollouts it cut raised RolloutFailed, verl drew fresh rows for 4 groups, and training went on.

A task whose episode ends unscored, a prompt that doesn't fit, and an agent loop run twice raise too. verl's V1 trainer logs each such session and leaves it out.

What verl reportsCopy Anchor Link

Each scored rollout returns reward_extra_info, which verl's V1 trainer averages over each eval under val-aux/<data_source>/, and writes per rollout to trainer.validation_data_dir when that's set:

step:0 - val-core/agentenv/gridsim-v1/reward/mean@1:0.5345 - val-aux/agentenv/gridsim-v1/score/grade/mean@1:0.5345 - val-aux/agentenv/gridsim-v1/attempts/mean@1:1.0 - val-aux/agentenv/gridsim-v1/model_seconds/mean@1:49.61 - val-aux/agentenv/gridsim-v1/env_seconds/mean@1:71.48 - …

GridSim's one verifier step is grade, so its score is the reward. With all 200 eval rollouts started at once and 64 sessions on the server, most of a rollout's 71 env seconds went to waiting for a session: 32 GridSim episodes at once spend about 25 seconds each in the env, 18 to 20 of them deploying, under a second a tool call and 4 scoring.

verl's V1 trainer doesn't report these per training step: it logs reward_extra_info per eval only, and its training dumps (trainer.rollout_data_dir) leave it out. A training step's own timing_s/ metrics show where its time goes. In an 8-step GridSim run a step took 75 seconds: 42 generating the 32 rollouts, 7 computing the old log-probs, 23 updating the actor and 2.5 loading the new weights into vLLM. Of the 42 seconds of generation, a rollout spent about 25 in the env, most of them deploying.

The async trainerCopy Anchor Link

verl's V1 trainer is synchronous by default (trainer.v1.trainer_mode=sync): a step plays its rollouts, waits for the slowest, then trains. With colocate_async, still on one GPU, verl keeps one more batch of rollouts going (trainer.v1.colocate_async.num_warmup_batches, 1 by default). A step trains once it has its batch; the rollouts still running pause their generation while the actor trains, then go on with the new weights. Their episodes stay open through the pause, so the agent loop needs no change.

The time saved is generation's: a step no longer waits for its slowest rollout, which finishes in the next one. The price is that each rollout an async step trains on was partly generated with the previous step's weights (training/off_policy/trajectory_staleness_worst is 1). Two batches in flight also hold twice the sessions, so set max_concurrent_envs to at least (1 + num_warmup_batches) × data.train_batch_size × rollout.n, 64 here, or the next batch's rollouts wait for sessions.

Multi-turn tasks and other tool-call formatsCopy Anchor Link

A task with several turns plays as one rollout. Each row offers end_turn with the task's tools: the model calls it with its answer, and its result is the next turn's prompt, which verl adds to the context like any tool answer. When the model replies without calling a tool, its reply ends the turn it left open, and any turns after it end empty. Tell the model in the system prompt to end each turn with end_turn. A task that records several scores needs a reward that combines them, given to build_app, here their mean:

app = build_app("bundles/my-bundle", reward=lambda scores: sum(scores.values()) / len(scores),
                max_concurrent_envs=64)

The agent loop gives verl's tool-call parsers each tool's typed parameters, which qwen3_coder needs to convert a call's arguments. A test bundle of four two-turn questions about an inbox, with a regex verifier a turn (answer-1 and answer-2), ran with Qwen3.5 and multi_turn.format=qwen3_coder:

step:0 - val-core/agentenv/inbox-turns/reward/mean@1:0.75 - val-aux/agentenv/inbox-turns/score/answer-1/mean@1:1.0 - val-aux/agentenv/inbox-turns/score/answer-2/mean@1:0.5 - val-aux/num_turns/mean:7.0 - …

Qwen3.5-2B scored 0.375 on the same tasks, 0.75 on the first turn and 0 on the second, in 4 turns every time: it answered the first question in a reply rather than with end_turn, so it never saw the second. After 5 GRPO steps through the agent loop it scored 0.5, all of it on the first turn. The reward alone doesn't show that; the scores by step do. The 2B run used model.use_remove_padding=False, actor.use_torch_compile=False and actor.entropy_from_logits_with_chunking=True; with them, training the 4B on one L40S ran out of memory.

What's supportedCopy Anchor Link

LimitsCopy Anchor Link

• An eval of failures stops the run. If every rollout of an eval failed, as when the env server is out of reach for longer than reset_timeout or every deploy fails, verl's V1 validation stops on the empty eval. verl doesn't tell an agent loop it's in an eval, so the loop can't avoid it.

• Scores per training step. verl's V1 trainer reports reward_extra_info per eval only, and GDPO's reward components read it only in verl's deprecated trainer.

• One deploy per rollout. Every rollout deploys its envs fresh, and the deploy counts in its time: a GridSim deploy took 18 to 20 seconds with 32 at once, close to half a step's generation.

• A server killed hard leaves its envs running. Stop it with SIGTERM, which tears its episodes down.

• Local Docker runs out of networks past about 30 episodes. With the local sandbox, each episode of an env that runs in a container gets a Docker network of its own, and Docker's default address pools hold about 30. More episodes at once fail to deploy with all predefined address pools have been fully subnetted. Add wider pools to Docker's daemon.json, and restart Docker before starting the server:

  Copy Text

  { "default-address-pools": [{ "base": "10.200.0.0/14", "size": 24 }] }