# verl (https://www.agentenvframework.com/docs/training/verl) > Train using verl ## How a rollout runs 1. verl’s agent-loop worker hands a row of the split’s parquet file to `AgentEnvAgentLoop`, which opens a session on the bundle’s OpenEnv server and starts the row’s task. 2. The server starts the task as an episode: it deploys the env, loads the task, and opens the agent’s turn, while verl’s vLLM writes the model’s first reply to the row’s prompt and tools. 3. verl parses each tool call with `multi_turn.format`’s parser, the loop calls the tool on the episode, and its answer joins the context, turn after turn up to `max_assistant_turns`. 4. The model’s last reply ends the turn it left open; a task with another turn answers `end_turn` with the next prompt. The task’s verifier scores the episode. 5. The score is the rollout’s `reward_score`, so verl calls no reward function; a rollout the env failed is played again, or left out of training. The episode is torn down. Parts of the scene: - **verl**: verl’s trainer runs each row through an agent loop in its agent-loop workers, generating on its own vLLM. The row’s `agent_name` picks `AgentEnvAgentLoop`, registered from `agent_loop.yaml`. verl owns the model, the generation and the update; the environment owns the reward. - **A dataset row**: A row of `.parquet`: the task’s prompt as chat messages, `agent_name: agentenv`, and in `extra_info` the `split`, `index` and `task_id` the loop plays, with the row’s tools. The prompt is yours to change; the task stays the same. - **AgentEnvAgentLoop**: verl’s `ToolAgentLoop` with the row’s tools in place of verl’s own, so verl’s tokenization, tool parsers and turn caps are its own. verl builds one per rollout; it holds one session, so one episode. - **The session**: One OpenEnv WebSocket session per rollout, waiting out a server at its limit or restarting. A rollout the env fails is played again on a new one, up to `attempts` times within `reset_timeout`. - **The OpenEnv server**: `agent_env.openenv.build_app(bundle)` serves every task of a bundle as an OpenEnv env, the split `all` plus one per eval. With `token=`, it can run on another machine than verl’s. - **The env**: The task’s env, deployed fresh for this episode in a sandbox, here a container on your Docker. Its tools are what the model calls. - **The agent’s turn**: The task prompts one agent, and your model plays it from outside: verl renders the turn’s prompt and tools with the model’s chat template, and each tool call the parser finds goes to the episode. - **The verifier**: The task’s own verifier step scores the episode once the turn ends, the same score an evaluation run records. With several scores, `reward=` picks the one to train on. - **The reward**: The episode’s reward is the rollout’s `reward_score`, and its scores, attempts and seconds its `reward_extra_info`. A rollout the env failed on every attempt raises, and verl’s V1 trainer leaves it out of the loss, the group’s baseline and the eval. [verl](https://github.com/volcengine/verl) plays multi-turn rollouts in agent loops: for each row, an agent loop generates on verl's own vLLM, parses the model's tool calls, adds their answers to the context, and returns the tokens and the reward. AgentEnv's agent loop plays each row's task as one episode on a bundle's OpenEnv server. Three pieces connect verl to a bundle: | Piece | What it does | | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `agent_env.openenv.build_app(bundle)` | Serves every task of a bundle as an [OpenEnv](https://www.agentenvframework.com/docs/openenv.md) env: the split `all`, plus one split per eval. | | `python -m agent_env.training.verl` | Writes the bundle's rows as verl's dataset, a parquet file per split, and `agent_loop.yaml`, which registers the agent loop with the server's URL. | | `agent_env.training.verl.loop.AgentEnvAgentLoop` | verl's `ToolAgentLoop` with each row's tools in place of verl's own: each call goes to the row's episode, and the episode's reward is the rollout's `reward_score`. | verl's tokenization, tool parsers, turn caps and truncation stay verl's own, and verl calls no reward function: the task's verifier scores each episode. ## Install verl runs on Linux with NVIDIA GPUs. This page's runs used one L40S, the trainer and vLLM sharing it. Run verl in its CUDA 13 image, whose uv cache holds every wheel its lockfile names, and install the framework into its environment: ```bash title="Terminal" docker run -d --name verl --gpus all --network host --ipc host --shm-size 32g \ -v "$PWD:$PWD" -w "$PWD" verlai/verl:uv.cu130.dev5 sleep infinity docker exec verl bash -c "cd /workspace/verl && uv sync --frozen --offline --extra vllm --extra fsdp" docker exec verl uv pip install --python /workspace/verl/.venv/bin/python 'agentenv-framework[verl]>=0.9.1401' ``` Run verl's commands with `docker exec verl /workspace/verl/.venv/bin/python ...`. The bundle's server runs where its envs deploy, with Docker running, in an environment of its own: ```bash title="Terminal" pip install 'agentenv-framework[openenv]>=0.9.1401' ``` > [!WARNING] > **Outside verl's image** > > verl 0.9.1's and main's `uv.lock` point flash-attn at a wheel that is no longer published, so > `uv sync --extra fsdp` fails outside the image. verl itself installs from PyPI (`pip install > verl==0.9.1`), which is how the framework's CI checks the agent loop, on CPU. ## Train on GridSim GridSim is the bundle the [TRL page](https://www.agentenvframework.com/docs/training/trl.md#train-on-gridsim) trains on. Each task runs one grid battery for one day in the ERCOT day-ahead market: the model reads the day's 24 hourly prices and submits a schedule, one integer an hour, to charge, sell or idle. The env grades the schedule against the day's exact optimum: 0 for nothing useful, 0.2 for idling all day, 1 for the optimum. The bundle has 2,000 train days and 200 eval days. ### Serve the bundle ```bash title="Terminal" hf download earakely-scale/GridSimEnv-AgentEnv --repo-type dataset --include "bundles/gridsim-v1/*" --local-dir . ``` ```python title="serve.py" import uvicorn from agent_env.openenv import build_app app = build_app("bundles/gridsim-v1", max_concurrent_envs=64) uvicorn.run(app, host="127.0.0.1", port=8200) ``` `max_concurrent_envs` covers a step's 32 rollouts, or the two batches [the async trainer](#the-async-trainer) keeps going; an eval's 200 rollouts wait their turn for a session. ### Write the rows ```bash title="Terminal" python -m agent_env.training.verl http://127.0.0.1:8200 data \ --tokenizer Qwen/Qwen3-4B --max-prompt-length 3072 --chat-template-kwargs '{"enable_thinking": false}' ``` ```text title="Output" wrote agent_loop.yaml, all.parquet, eval.parquet, train.parquet to data the longest prompt is 1169 tokens with its tools ``` The command plays one task of each toolset to list the tools its rows offer, so it deploys an env once. Each row holds the task's prompt as chat messages, `agent_name: agentenv`, and in `extra_info` the split, index and task id the agent loop plays, with the row's tools. `--tokenizer` counts each row's prompt with its tools as the model's chat template renders them, with verl's `data.apply_chat_template_kwargs` given as `--chat-template-kwargs`. A row over `--max-prompt-length` is refused before anything is written: verl would cut it from the left, losing the system prompt and the tools, and verl's own length filter doesn't count a row's tools. ```yaml title="data/agent_loop.yaml" - name: agentenv _target_: agent_env.training.verl.loop.AgentEnvAgentLoop url: http://127.0.0.1:8200 tool_timeout: 60.0 reset_timeout: 600.0 attempts: 3 ``` ### Warm start with SFT Played as it comes, Qwen3-4B idles: it scores 0.18 on the 200 eval days. Warm-start it with the TRL page's [`sft.py`](https://www.agentenvframework.com/docs/training/trl.md#warm-start-with-sft), in an environment with TRL, so it saves `qwen3-4b-gridsim-sft`. It took 25 minutes on one L40S. ### Train with GRPO This run uses the TRL, NeMo RL and prime-rl runs' settings: the same SFT warm start, LoRA r32/α64 on every linear layer, learning rate 2e-5 cosine with 3 warmup steps, 32 rollouts a step (4 days × 8) at temperature 1.1, thinking off, and at most four model turns a rollout. The parts that are about AgentEnv: ```bash title="verl_gridsim.sh (excerpt)" python -m verl.trainer.main_ppo \ data.train_files=data/train.parquet data.val_files=data/eval.parquet \ data.return_raw_chat=True +data.apply_chat_template_kwargs.enable_thinking=False \ actor_rollout_ref.rollout.multi_turn.enable=True \ actor_rollout_ref.rollout.multi_turn.format=hermes \ actor_rollout_ref.rollout.multi_turn.max_assistant_turns=4 \ actor_rollout_ref.rollout.multi_turn.max_tool_response_length=8192 \ actor_rollout_ref.rollout.agent.agent_loop_config_path=data/agent_loop.yaml \ actor_rollout_ref.rollout.agent.default_agent_loop=agentenv \ actor_rollout_ref.rollout.layered_summon=False \ trainer.v1.sampler.sync_refill_failed_groups=True \ ... ``` The whole command, as it ran, is [`verl_gridsim.sh`](/training/verl_gridsim.sh). Run it in verl's environment from the folder holding `qwen3-4b-gridsim-sft` and `data/`. It evaluates on the 200 eval days before the first step and every 20 steps, one rollout a day at the training temperature. ```text title="Output (step 21 of the run, trimmed)" step:21 - critic/score/mean:0.6845 - response_length/mean:815.75 - num_turns/mean:7.94 - timing_s/gen:32.77 - timing_s/step:63.01 - … ``` `critic/score/mean` is the mean of the step's 32 rewards. verl counts the user and tool turns too, so eight turns are the four model turns the cap allows. ### What it learns | Qwen3-4B | eval reward (200 days) | |---|---| | after SFT (GRPO step 0) | 0.522 | | GRPO step 20 | 0.588 | | GRPO step 40 | 0.600 | Mean train reward of each GRPO step's 32 rollouts, by step: 1: 0.620, 2: 0.638, 3: 0.438, 4: 0.670, 5: 0.585, 6: 0.692, 7: 0.516, 8: 0.574, 9: 0.615, 10: 0.533, 11: 0.581, 12: 0.665, 13: 0.609, 14: 0.281, 15: 0.521, 16: 0.731, 17: 0.761, 18: 0.481, 19: 0.657, 20: 0.567, 21: 0.731, 22: 0.684, 23: 0.520, 24: 0.632, 25: 0.669, 26: 0.756, 27: 0.664, 28: 0.589, 29: 0.696, 30: 0.601, 31: 0.834, 32: 0.544, 33: 0.830, 34: 0.585, 35: 0.706, 36: 0.892, 37: 0.589, 38: 0.686, 39: 0.777, 40: 0.886. | Qwen3-4B, eval reward on the 200 eval days | verl | prime-rl | NeMo RL | TRL | | ------------------------------------------ | ---- | -------- | ------- | ---- | | After SFT (GRPO step 0) | 0.52 | 0.56 | 0.55 | 0.57 | | After 20 GRPO steps | 0.59 | 0.58 | 0.68 | 0.56 | | After 40 GRPO steps | 0.60 | 0.57 | 0.67 | 0.61 | All four start from the same SFT model and score it alike, within what one rollout a day can tell apart (about ±0.06). Through verl the train reward rises from 0.59 over the first 10 steps to 0.69 over the last 20, and the eval reward from 0.52 to 0.60; a second run, with the envs on the GPU machine, scored 0.52, 0.59 and 0.60. GridSim's greedy rule scores 0.76 on these days. This is one run of each trainer, so read the gaps as pointers, not a ranking. The 40 GRPO steps with the three evals took 67 minutes on one L40S: about 75 seconds a step, a little over half of it generation. ## Settings verl needs | Setting | Value here | Without it | | ---------------------------------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `rollout.multi_turn.format` | `hermes` | The model's tool-call format, which verl's parser reads: `hermes` for Qwen3, `qwen3_coder` for Qwen3.5 and Qwen3-Coder. A wrong one finds no tool calls, and every rollout ends after its first reply. | | `rollout.agent.agent_loop_config_path` | `data/agent_loop.yaml` | verl doesn't know the agent loop the rows name. | | `rollout.multi_turn.max_tool_response_length` | 8192 | verl cuts each tool answer to 256 characters; the agent loop warns once a tool when it does. | | `rollout.multi_turn.max_assistant_turns` | 4 | verl caps no turns, and a rollout runs until its response length runs out. | | `trainer.v1.sampler.sync_refill_failed_groups` | `True` | A step whose rollouts all failed stops the run instead of drawing fresh rows. | | `rollout.layered_summon` | `False` | With LoRA on one GPU, FSDP refuses the CPU offload layered summoning takes. | | `data.max_prompt_length` | 3072 | Over the writer's longest prompt; the agent loop refuses a prompt that doesn't fit. | ## Envs on another machine verl's agent-loop workers can run on any node of its Ray cluster, and each opens its sessions on the URL in `agent_loop.yaml`, so the bundle's server must be one every node reaches. Give it a token, which every request and WebSocket but `/health` must then carry: ```python title="serve.py" app = build_app("bundles/gridsim-v1", max_concurrent_envs=64, token=os.environ["AGENTENV_TOKEN"]) uvicorn.run(app, host="0.0.0.0", port=8200) ``` ```bash title="Terminal" python -m agent_env.training.verl http://envs.internal:8200 data --token-var AGENTENV_TOKEN ... ``` `--token-var` names the environment variable that holds the token: the writer reads it, and `agent_loop.yaml` records its name, never the token, for the agent loop to read in each worker. Set it where verl starts, which Ray workers inherit, or in `ray_kwargs.ray_init.runtime_env.env_vars`. Without a token, anyone who reaches the server deploys and drives its envs. The server speaks plain HTTP and WebSockets, so the token crosses the network unencrypted. Keep the server on a private network that only verl's nodes reach, as the run plotted above did, or put a TLS proxy in front of it and give the writer its `https://` URL, which the agent loop's sessions open as `wss://`. The run plotted above served GridSim this way, from a CPU machine of its own in the same network as the GPU one, and scored 0.52, 0.59 and 0.60 at steps 0, 20 and 40. ## Failed rollouts A rollout the env fails, a deploy, a call past its deadline or a server that went away, is played again from the start on a new episode, a fresh sample of the same policy: up to `attempts` times, but none starts once `reset_timeout` has passed since the rollout began. One the env fails on every attempt raises `RolloutFailed`, and verl's V1 trainer, the default, records the session as failed and leaves the rollout out of the loss, its group's baseline and the eval reward. A step left with a group of no rollouts at all draws a fresh row for it with `sync_refill_failed_groups`. Opening its session, a rollout waits for a server at its limit or one restarting, within `reset_timeout`, so a restart delays rollouts rather than failing them. In a test run the env server was restarted twice while rollouts opened their sessions; those steps ran up to 54 seconds longer and lost no rollout. In another, with one attempt a rollout and the server stopped mid-episode, the rollouts it cut raised `RolloutFailed`, verl drew fresh rows for 4 groups, and training went on. A task whose episode ends unscored, a prompt that doesn't fit, and an agent loop run twice raise too. verl's V1 trainer logs each such session and leaves it out. ## What verl reports Each scored rollout returns `reward_extra_info`, which verl's V1 trainer averages over each eval under `val-aux//`, and writes per rollout to `trainer.validation_data_dir` when that's set: | Key | What it holds | | ----------------- | ----------------------------------------------------------------------------------------------------------- | | `score/` | Each score the task's verifiers recorded, by the id of the step that recorded it. The reward combines them. | | `attempts` | How many times the rollout was played: 1 unless the env failed it. | | `model_seconds` | Its time generating on verl's vLLM. | | `env_seconds` | Its time in the env: waiting for a session, deploying, calling tools, ending turns and scoring. | ```text title="Output (the step-0 eval of a GridSim run, trimmed)" step:0 - val-core/agentenv/gridsim-v1/reward/mean@1:0.5345 - val-aux/agentenv/gridsim-v1/score/grade/mean@1:0.5345 - val-aux/agentenv/gridsim-v1/attempts/mean@1:1.0 - val-aux/agentenv/gridsim-v1/model_seconds/mean@1:49.61 - val-aux/agentenv/gridsim-v1/env_seconds/mean@1:71.48 - … ``` GridSim's one verifier step is `grade`, so its score is the reward. With all 200 eval rollouts started at once and 64 sessions on the server, most of a rollout's 71 env seconds went to waiting for a session: 32 GridSim episodes at once spend about 25 seconds each in the env, 18 to 20 of them deploying, under a second a tool call and 4 scoring. verl's V1 trainer doesn't report these per training step: it logs `reward_extra_info` per eval only, and its training dumps (`trainer.rollout_data_dir`) leave it out. A training step's own `timing_s/` metrics show where its time goes. In an 8-step GridSim run a step took 75 seconds: 42 generating the 32 rollouts, 7 computing the old log-probs, 23 updating the actor and 2.5 loading the new weights into vLLM. Of the 42 seconds of generation, a rollout spent about 25 in the env, most of them deploying. ## The async trainer verl's V1 trainer is synchronous by default (`trainer.v1.trainer_mode=sync`): a step plays its rollouts, waits for the slowest, then trains. With `colocate_async`, still on one GPU, verl keeps one more batch of rollouts going (`trainer.v1.colocate_async.num_warmup_batches`, 1 by default). A step trains once it has its batch; the rollouts still running pause their generation while the actor trains, then go on with the new weights. Their episodes stay open through the pause, so the agent loop needs no change. | GridSim, 8 GRPO steps on one L40S | `sync` | `colocate_async` | | --------------------------------- | ------ | ---------------- | | Step, median | 75 s | 66 s | | Generation in a step, median | 42.7 s | 32.8 s | | All 8 steps | 651 s | 531 s | | Train reward, mean of the 8 steps | 0.58 | 0.61 | | Failed or replayed rollouts | 0 | 0 | The time saved is generation's: a step no longer waits for its slowest rollout, which finishes in the next one. The price is that each rollout an async step trains on was partly generated with the previous step's weights (`training/off_policy/trajectory_staleness_worst` is 1). Two batches in flight also hold twice the sessions, so set `max_concurrent_envs` to at least `(1 + num_warmup_batches) × data.train_batch_size × rollout.n`, 64 here, or the next batch's rollouts wait for sessions. ## Multi-turn tasks and other tool-call formats A task with several turns plays as one rollout. Each row offers `end_turn` with the task's tools: the model calls it with its answer, and its result is the next turn's prompt, which verl adds to the context like any tool answer. When the model replies without calling a tool, its reply ends the turn it left open, and any turns after it end empty. Tell the model in the system prompt to end each turn with `end_turn`. A task that records several scores needs a reward that combines them, given to `build_app`, here their mean: ```python title="serve.py" app = build_app("bundles/my-bundle", reward=lambda scores: sum(scores.values()) / len(scores), max_concurrent_envs=64) ``` The agent loop gives verl's tool-call parsers each tool's typed parameters, which `qwen3_coder` needs to convert a call's arguments. A test bundle of four two-turn questions about an inbox, with a regex verifier a turn (`answer-1` and `answer-2`), ran with Qwen3.5 and `multi_turn.format=qwen3_coder`: ```text title="Output (Qwen3.5-4B, the step-0 eval, trimmed)" step:0 - val-core/agentenv/inbox-turns/reward/mean@1:0.75 - val-aux/agentenv/inbox-turns/score/answer-1/mean@1:1.0 - val-aux/agentenv/inbox-turns/score/answer-2/mean@1:0.5 - val-aux/num_turns/mean:7.0 - … ``` Qwen3.5-2B scored 0.375 on the same tasks, 0.75 on the first turn and 0 on the second, in 4 turns every time: it answered the first question in a reply rather than with `end_turn`, so it never saw the second. After 5 GRPO steps through the agent loop it scored 0.5, all of it on the first turn. The reward alone doesn't show that; the scores by step do. The 2B run used `model.use_remove_padding=False`, `actor.use_torch_compile=False` and `actor.entropy_from_logits_with_chunking=True`; with them, training the 4B on one L40S ran out of memory. ## What's supported | | | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | verl | 0.9.1 and main, with the V1 trainer. The page's runs used verl main in `verlai/verl:uv.cu130.dev5`; a CI job plays rows through the agent loop on both every week. | | Tool-call formats | Any `multi_turn.format` verl parses: the agent loop gives verl's parsers each tool's typed parameters. | | Multi-turn tasks | `end_turn` hands the model each next prompt. | | Bundles with several toolsets | Each row lists its own task's tools. | | Envs | Wherever the bundle's server deploys them: `build_app(..., sandbox=)` takes any [sandbox provider](https://www.agentenvframework.com/docs/sandboxes.md). | ## Limits * **An eval of failures stops the run.** If every rollout of an eval failed, as when the env server is out of reach for longer than `reset_timeout` or every deploy fails, verl's V1 validation stops on the empty eval. verl doesn't tell an agent loop it's in an eval, so the loop can't avoid it. * **Scores per training step.** verl's V1 trainer reports `reward_extra_info` per eval only, and GDPO's reward components read it only in verl's deprecated trainer. * **One deploy per rollout.** Every rollout deploys its envs fresh, and the deploy counts in its time: a GridSim deploy took 18 to 20 seconds with 32 at once, close to half a step's generation. * **A server killed hard leaves its envs running.** Stop it with SIGTERM, which tears its episodes down. * **Local Docker runs out of networks past about 30 episodes.** With the `local` sandbox, each episode of an env that runs in a container gets a Docker network of its own, and Docker's default address pools hold about 30. More episodes at once fail to deploy with `all predefined address pools have been fully subnetted`. Add wider pools to Docker's `daemon.json`, and restart Docker before starting the server: ```json title="/etc/docker/daemon.json" { "default-address-pools": [{ "base": "10.200.0.0/14", "size": 24 }] } ```