# SkyRL (https://www.agentenvframework.com/docs/training/skyrl)

> Train using SkyRL

## How a rollout runs

1. SkyRL’s generator builds an `AgentEnvTextEnv` for a row of the split’s parquet file, and its `init` opens a session on the bundle’s OpenEnv server and starts the row’s task.
2. The server starts the task as an episode: it deploys the env, loads the task, and opens the agent’s turn, while SkyRL’s vLLM writes the model’s first reply to the row’s prompt, its tools in the system prompt.
3. Each `step` parses the reply’s tool calls in the row’s `tool_format`, runs them on the episode, and returns their answers as tool messages, turn after turn up to `generator.max_turns`.
4. A reply without a tool call ends the turn it left open; a task with another turn answers `end_turn` with the next prompt. The task’s verifier scores the episode.
5. The score is the last step’s reward, so SkyRL calls no reward function; a rollout the env failed is played again, or its group left out of training. The episode is torn down.

Parts of the scene:

- **SkyRL**: SkyRL’s trainer generates each rollout in its generator, on its own vLLM, building the env the row’s `env_class` names. `python -m agent_env.training.skyrl.train` registers `agentenv` and generates with `AgentEnvGenerator`. SkyRL owns the model, the generation and the update; the environment owns the reward.
- **A dataset row**: A row of `<split>.parquet`: the task’s prompt as chat messages with its tools in the system prompt, as the model’s chat template renders them, `env_class: agentenv`, the `split`, `index` and `task_id` the env plays, and the server’s settings. The prompt is yours to change; the task stays the same.
- **AgentEnvTextEnv**: A skyrl-gym `BaseTextEnv` per rollout: each step runs the reply’s tool calls on the row’s episode and answers with their results. SkyRL’s tokenization and turn caps are its own; the env reads the calls, in `hermes` or `qwen3_coder`.
- **The session**: One OpenEnv WebSocket session per rollout, waiting out a server at its limit or restarting. A rollout the env fails is played again on a new one, up to `attempts` times within `reset_timeout`.
- **The OpenEnv server**: `agent_env.openenv.build_app(bundle)` serves every task of a bundle as an OpenEnv env, the split `all` plus one per eval. With `token=`, it can run on another machine than SkyRL’s.
- **The env**: The task’s env, deployed fresh for this episode in a sandbox, here a container on your Docker. Its tools are what the model calls.
- **The agent’s turn**: The task prompts one agent, and your model plays it from outside: SkyRL renders the conversation with the model’s chat template, the tools already in its system prompt, and each tool call the env reads goes to the episode.
- **The verifier**: The task’s own verifier step scores the episode once the turn ends, the same score an evaluation run records. With several scores, `reward=` picks the one to train on.
- **The reward**: The episode’s reward is the last step’s, and its scores, attempts and tool calls the env’s metrics, which SkyRL logs every step. A rollout the env failed on every attempt leaves its whole group out of the loss and the baseline.

[SkyRL](https://github.com/NovaSky-AI/SkyRL) plays multi-turn rollouts in its generator: for each
row, it builds the env the row's `env_class` names, generates on its own vLLM, and hands each reply
to the env's `step`, whose answer joins the context until the env says the episode is done. AgentEnv's
env plays each row's task as one episode on a bundle's OpenEnv server. Four pieces connect SkyRL to a
bundle:

| Piece                                          | What it does                                                                                                                                                            |
| ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `agent_env.openenv.build_app(bundle)`          | Serves every task of a bundle as an [OpenEnv](https://www.agentenvframework.com/docs/openenv.md) env: the split `all`, plus one split per eval.                                                             |
| `python -m agent_env.training.skyrl`           | Writes the bundle's rows as SkyRL's dataset, a parquet file per split, each row with its tools in its system prompt and the server's settings.                          |
| `agent_env.training.skyrl.env.AgentEnvTextEnv` | A skyrl-gym `BaseTextEnv`, registered as `agentenv`: each step runs the reply's tool calls on the row's episode, and the step that ends the episode returns its reward. |
| `python -m agent_env.training.skyrl.train`     | SkyRL's trainer with the env registered and `AgentEnvGenerator` generating, which plays a rollout the env fails again.                                                  |

SkyRL's tokenization, turn cap and training stay SkyRL's own, and it calls no reward function: the
task's verifier scores each episode. SkyRL's generator shows the model no tools and reads no tool
calls, so the writer puts the tools in each row's system prompt and the env reads the calls.

## Install

SkyRL runs on Linux with NVIDIA GPUs. This page's runs used one L40S, the trainer and vLLM sharing
it. Run SkyRL from its repository in its CUDA 13 image, and install the framework into its
environment:

```bash title="Terminal"
docker run -d --name skyrl --gpus all --network host --ipc host --shm-size 32g \
  -v "$PWD:$PWD" -w "$PWD" novaskyai/skyrl-train-ray-2.57.0-py3.12-cu13.0 sleep infinity
docker exec skyrl bash -c "git clone https://github.com/NovaSky-AI/SkyRL && cd SkyRL && uv sync --extra fsdp"
docker exec skyrl bash -c "cd SkyRL && uv pip install --python .venv/bin/python 'agentenv-framework[skyrl]>=0.9.1404'"
```

Run SkyRL's commands with `docker exec skyrl SkyRL/.venv/bin/python ...`. The bundle's server runs
where its envs deploy, with Docker running, in an environment of its own:

```bash title="Terminal"
pip install 'agentenv-framework[openenv]>=0.9.1404'
```

> [!WARNING]
> **Training as a login user on Ubuntu**
>
> vLLM and PyTorch's data loader share memory between processes. On Ubuntu, systemd-logind deletes a
> user's shared memory when that user's last login session ends (`RemoveIPC=yes`), so a run started
> as a login user, in a container sharing the host's IPC, fails once that user logs out: vLLM can't
> find its ring buffer, or a data loader worker its semaphore. Run as root or the image's own user, or
> set `RemoveIPC=no` in `/etc/systemd/logind.conf` and restart `systemd-logind`.

## Train on GridSim

GridSim is the bundle the [TRL page](https://www.agentenvframework.com/docs/training/trl.md#train-on-gridsim) trains on. Each task runs
one grid battery for one day in the ERCOT day-ahead market: the model reads the day's 24 hourly
prices and submits a schedule, one integer an hour, to charge, sell or idle. The env grades the
schedule against the day's exact optimum: 0 for nothing useful, 0.2 for idling all day, 1 for the
optimum. The bundle has 2,000 train days and 200 eval days.

### Serve the bundle

```bash title="Terminal"
hf download earakely-scale/GridSimEnv-AgentEnv --repo-type dataset --include "bundles/gridsim-v1/*" --local-dir .
```

```python title="serve.py"
import uvicorn

from agent_env.openenv import build_app

app = build_app("bundles/gridsim-v1", max_concurrent_envs=64)
uvicorn.run(app, host="127.0.0.1", port=8200)
```

`max_concurrent_envs` covers a step's 32 rollouts; an eval's 200 rollouts play 64 at a time.

### Write the rows

```bash title="Terminal"
python -m agent_env.training.skyrl http://127.0.0.1:8200 data \
  --tokenizer Qwen/Qwen3-4B --max-prompt-length 3072 --chat-template-kwargs '{"enable_thinking": false}'
```

```text title="Output"
wrote all.parquet, eval.parquet, train.parquet to data
the longest prompt is 1169 tokens with its tools
```

The command plays one task of each toolset to list the tools its rows offer, so it deploys an env
once. Each row holds the task's prompt as chat messages, `env_class: agentenv`, the split, index and
task id the env plays with the row's tools in `episode`, and the server's settings in `server`.

SkyRL's generator passes the chat template no tools, so `--tokenizer` puts each row's tools in its
system prompt as the model's chat template renders them with `tools=`, and checks the prompt renders
the same text either way: the model sees its tools as it would anywhere else. A template that renders
tools outside the system turn is refused. `--chat-template-kwargs` is SkyRL's
`generator.chat_template_kwargs`. A row over `--max-prompt-length` is refused before anything is
written, counted both as SkyRL's prompt filter counts it, without the template settings, and as its
generator renders it, with them; the filter would drop it without saying.

`--tool-format` names the format the template teaches the model to call tools in, which the env
reads: `hermes`, the default, for Qwen2.5 and Qwen3, and `qwen3_coder` for Qwen3.5 and Qwen3-Coder.

### Warm start with SFT

Played as it comes, Qwen3-4B idles: it scores 0.18 on the 200 eval days. Warm-start it with the
TRL page's [`sft.py`](https://www.agentenvframework.com/docs/training/trl.md#warm-start-with-sft), in an environment with TRL, so it saves
`qwen3-4b-gridsim-sft`. It took 26 minutes on one L40S.

### Train with GRPO

This run uses the other trainers' runs' settings: the same SFT warm start, LoRA r32/α64 on every
linear layer, learning rate 2e-5 cosine with 3 warmup steps, 32 rollouts a step (4 days × 8) at
temperature 1.1, thinking off, and at most four model turns a rollout. The parts that are about
AgentEnv:

```bash title="skyrl_gridsim.sh (excerpt)"
python -m agent_env.training.skyrl.train \
  data.train_data="['data/train.parquet']" data.val_data="['data/eval.parquet']" \
  environment.env_class=agentenv environment.skyrl_gym.max_env_workers=64 \
  generator.batched=false generator.max_turns=4 \
  generator.chat_template_kwargs.enable_thinking=false \
  generator.max_input_length=8192 trainer.max_prompt_length=3072 \
  ...
```

The whole command, as it ran, is [`skyrl_gridsim.sh`](/training/skyrl_gridsim.sh). Run it in
SkyRL's environment from the folder holding `qwen3-4b-gridsim-sft` and `data/`. It evaluates on the
200 eval days before the first step and every 20 steps, one rollout a day at the training
temperature.

```text title="Output (a step of a short run on the same rows, trimmed)"
Step 4:
{'environment/avg_attempts': '1.0000',
 'environment/avg_failed': '0.0000',
 'environment/avg_score/grade': '0.4924',
 'environment/avg_tool_calls': '2.8125',
 'environment/avg_unparsed_calls': '0.1250',
 'generate/trajectory_time_env_mean': '20.7275',
 'generate/trajectory_time_llm_mean': '13.9976',
 'reward/avg_raw_reward': '0.4924',
 'timing/step': '76.3410',
 …}
```

`reward/avg_raw_reward` is the mean of the step's 32 rewards, here GridSim's one score,
`environment/avg_score/grade`. The env's other metrics are under [What SkyRL
reports](#what-skyrl-reports).

### What it learns

| Qwen3-4B | eval reward (200 days) |
|---|---|
| after SFT (GRPO step 0) | 0.536 |
| GRPO step 20 | 0.549 |
| GRPO step 40 | 0.614 |

Mean train reward of each GRPO step's 32 rollouts, by step: 1: 0.579, 2: 0.593, 3: 0.358, 4: 0.474, 5: 0.286, 6: 0.456, 7: 0.668, 8: 0.775, 9: 0.731, 10: 0.586, 11: 0.684, 12: 0.829, 13: 0.706, 14: 0.569, 15: 0.483, 16: 0.590, 17: 0.713, 18: 0.590, 19: 0.743, 20: 0.715, 21: 0.447, 22: 0.739, 23: 0.612, 24: 0.446, 25: 0.711, 26: 0.855, 27: 0.649, 28: 0.750, 29: 0.648, 30: 0.908, 31: 0.637, 32: 0.547, 33: 0.728, 34: 0.761, 35: 0.665, 36: 0.610, 37: 0.655, 38: 0.575, 39: 0.737, 40: 0.531.

| Qwen3-4B, eval reward on the 200 eval days | SkyRL | verl | prime-rl | NeMo RL | TRL  |
| ------------------------------------------ | ----- | ---- | -------- | ------- | ---- |
| After SFT (GRPO step 0)                    | 0.54  | 0.52 | 0.56     | 0.55    | 0.57 |
| After 20 GRPO steps                        | 0.55  | 0.59 | 0.58     | 0.68    | 0.56 |
| After 40 GRPO steps                        | 0.61  | 0.60 | 0.57     | 0.67    | 0.61 |

All five start from the same SFT model and score it alike, within what one rollout a day can tell
apart (about ±0.06). Through SkyRL the train reward rises from 0.55 over the first 10 steps to 0.66
over the last 20, and the eval reward from 0.54 to 0.61. GridSim's greedy rule scores 0.76 on these
days. This is one run of each trainer, so read the gaps as pointers, not a ranking.

The 40 GRPO steps with the three evals took 62 minutes on one L40S: about 70 seconds a step, a little
over half of it generation. No rollout failed or was played again.

## Settings SkyRL needs

| Setting                                 | Value here               | Without it                                                                                                                                                   |
| --------------------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `generator.batched`                     | `false`                  | SkyRL's batched generation plays one turn without the env's steps; `AgentEnvGenerator` refuses it.                                                           |
| `generator.max_turns`                   | 4                        | The model turns a rollout may take; SkyRL's default of 1 ends every episode after the model's first reply.                                                   |
| `environment.skyrl_gym.max_env_workers` | 64                       | SkyRL's threads for envs, and so the rollouts that play at once: a rollout waiting on the server holds its thread. Match the server's `max_concurrent_envs`. |
| `generator.chat_template_kwargs`        | `enable_thinking: false` | The same settings the writer rendered the rows with.                                                                                                         |
| `generator.max_input_length`            | 8192                     | The context a rollout may grow to; SkyRL ends one that outgrows it, and the episode scores 0.                                                                |
| `trainer.max_prompt_length`             | 3072                     | Over the writer's longest prompt; SkyRL's filter drops a longer row.                                                                                         |

## Envs on another machine

SkyRL's generator runs in a Ray task on any node of its cluster, and each env opens its session on
the URL in its row, so the bundle's server must be one that node reaches. Give it a token, which
every request and WebSocket but `/health` must then carry:

```python title="serve.py"
import os

import uvicorn

from agent_env.openenv import build_app

app = build_app("bundles/gridsim-v1", max_concurrent_envs=64, token=os.environ["AGENTENV_TOKEN"])
uvicorn.run(app, host="0.0.0.0", port=8200)
```

```bash title="Terminal"
python -m agent_env.training.skyrl http://envs.internal:8200 data --token-var AGENTENV_TOKEN ...
```

`--token-var` names the environment variable that holds the token: the writer reads it, and the rows
record its name, never the token. `agent_env.training.skyrl.train` reads it where it starts and hands
it to the Ray task that generates, on whichever node it runs. Without a token, anyone who reaches the
server deploys and drives its envs.

The server speaks plain HTTP and WebSockets, so the token crosses the network unencrypted. Keep the
server on a private network that only SkyRL's nodes reach, or put a TLS proxy in front of it and give
the writer its `https://` URL, which the env's sessions open as `wss://`.

## Failed rollouts

A rollout the env fails, a deploy, a call past its deadline or a server that went away, is played
again from the start on a new episode, a fresh sample of the same policy: up to `attempts` times, but
none starts once `reset_timeout` has passed since the rollout began. SkyRL can't leave one rollout
out of a step, and stops the step on an env's error, so `AgentEnvGenerator` leaves out the rollout's
whole group instead, as SkyRL's own Harbor integration does with a failed trial: in training, it
zeroes the group's loss masks and rewards, so the group is out of the gradient and its baseline. In
an eval, the failed rollout scores 0.

Opening its session, a rollout waits for a server at its limit or one restarting, within
`reset_timeout`, so a restart delays rollouts rather than failing them.

In two short runs on the GridSim rows, the env server was stopped while all 32 of a step's rollouts
were mid-episode, and was back 40 to 50 seconds later:

|                                          | `attempts: 3`                                     | `attempts: 1`                                         |
| ---------------------------------------- | ------------------------------------------------- | ----------------------------------------------------- |
| The step's rollouts                      | played again once the server was back, and scored | failed, and their groups left out                     |
| `environment/avg_attempts`, `avg_failed` | 2, 0                                              | 1, 1                                                  |
| The step                                 | 133 seconds rather than about 80                  | trained on none of them (`loss/avg_raw_advantages` 0) |
| The steps after                          | as before                                         | as before                                             |

A task whose episode ends unscored, or that offers a tool its row doesn't list, raises, and stops the
step: no replay changes it.

## What SkyRL reports

The env's metrics, whose means over each step's rollouts and each eval's SkyRL logs as
`environment/avg_<key>` and `eval/all/environment/avg_<key>`. They're named as SkyRL names a mean, so
where SkyRL joins groups' metrics, as its fully async trainer does, it averages them again rather than
adding them up:

| Key               | What it holds                                                                                               |
| ----------------- | ----------------------------------------------------------------------------------------------------------- |
| `score/<step id>` | Each score the task's verifiers recorded, by the id of the step that recorded it. The reward combines them. |
| `attempts`        | How many times the rollout was played: 1 unless the env failed it.                                          |
| `failed`          | Whether the env failed it on every attempt, which leaves its group out.                                     |
| `tool_calls`      | The tool calls the env ran.                                                                                 |
| `unparsed_calls`  | The tool calls it couldn't read, answered with why.                                                         |

SkyRL's own `generate/trajectory_time_llm_*` and `generate/trajectory_time_env_*` split each
rollout's time between the model and the env.

In the GridSim run a step took 70 seconds: 39 generating the 32 rollouts, 7 computing the old
log-probs, 20 updating the actor and 3 loading the new weights into vLLM. A rollout spent 17 seconds
in the env, its deploy included, and 11 in the model. An eval's 200 rollouts, 64 at a time, took 3
minutes.

## The fully async trainer

SkyRL's fully async trainer (`trainer.fully_async.enabled=true`) generates and trains at once, on GPUs
of their own (`trainer.placement.colocate_all=false`): rollouts go on playing while the policy trains,
on weights at most `trainer.fully_async.max_staleness_steps` behind. `agent_env.training.skyrl.train`
picks it from that setting, and the env and generator need nothing else.

| GridSim, 8 GRPO steps on 4 A10Gs, 2 training and 2 generating | sync   | fully async |
| ------------------------------------------------------------- | ------ | ----------- |
| Step, median                                                  | 69.6 s | 40.5 s      |
| All 8 steps                                                   | 562 s  | 402 s       |
| Rollouts a step behind the policy, after the first step       | none   | all         |
| Failed or replayed rollouts                                   | 0      | 0           |

Both used SkyRL's own example's settings for the async trainer: `max_staleness_steps=1`,
`num_parallel_generation_workers=8` (`train_batch_size × (max_staleness_steps + 1)`), and its
off-policy correction (`trainer.algorithm.policy_loss_type=rollout_is` with geometric sequence
masking). The model was Qwen3-4B without the warm start, which idles toward 0.2 in either run; the
comparison is of time, not of learning.

## Multi-turn tasks and other tool-call formats

A task with several turns plays as one rollout. Each row offers `end_turn` with the task's tools: the
model calls it with its answer, and its result is the next turn's prompt, which the env returns like
any tool answer. When the model replies without calling a tool, its reply ends the turn it left open,
and any turns after it end empty. Tell the model in the system prompt to end each turn with
`end_turn`. A task that records several scores needs a reward that combines them, given to
`build_app`, here their mean:

```python title="serve.py"
app = build_app("bundles/my-bundle", reward=lambda scores: sum(scores.values()) / len(scores),
                max_concurrent_envs=64)
```

For a model whose template teaches `qwen3_coder`, write the rows with `--tool-format qwen3_coder`;
the env types each argument by its tool's JSON Schema. SkyRL loads Qwen3.5 as a vision-language
model, so training it needs `trainer.remove_microbatch_padding=false`. A test bundle of four two-turn
questions about an inbox, with a regex verifier a turn (`answer-1` and `answer-2`), ran on Qwen3.5-2B
for 5 GRPO steps of 32 rollouts, about 2 minutes a step:

```text title="Output (Qwen3.5-2B, a rollout of step 5, trimmed)"
<tool_call>
<function=search_emails>
<parameter=query>
invoice
</parameter>
</function>
</tool_call><|im_end|>
<|im_start|>user
<tool_response>
[{"id": "m-1", "from": "billing@acme-supplies.example", "subject": "Invoice INV-1042 for October", …}]
</tool_response><|im_end|>
<|im_start|>assistant
…
The invoice number in your inbox is INV-1042.<|im_end|>
```

The env read every call the model made, none unparsed. The first turn's score rose from 0.78 over
step 1's rollouts to 0.97 over step 5's, and the eval from 0.375 to 0.5, all of it on the first turn:
like the rollout above, the model answered the first question in a reply rather than with `end_turn`,
so it never saw the second, as it did under [verl](https://www.agentenvframework.com/docs/training/verl.md#multi-turn-tasks-and-other-tool-call-formats).
The reward alone doesn't show that; `environment/avg_score/answer-2`, 0 at every step, does.

## What's supported

|                               |                                                                                                                                                                            |
| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| SkyRL                         | 0.4.0 and main. The page's runs used SkyRL main in `novaskyai/skyrl-train-ray-2.57.0-py3.12-cu13.0`; a CI job plays rows through the env and generator on both every week. |
| Tool-call formats             | `hermes` (Qwen2.5, Qwen3) and `qwen3_coder` (Qwen3.5, Qwen3-Coder).                                                                                                        |
| Multi-turn tasks              | `end_turn` hands the model each next prompt.                                                                                                                               |
| Bundles with several toolsets | Each row lists its own task's tools.                                                                                                                                       |
| Envs                          | Wherever the bundle's server deploys them: `build_app(..., sandbox=)` takes any [sandbox provider](https://www.agentenvframework.com/docs/sandboxes.md).                                                       |

## Limits

* **A failed rollout leaves its group out.** SkyRL has no way to leave out one rollout, so the
  rollouts of its group are left out with it.
* **A rollout that outgrows `generator.max_input_length` scores 0.** SkyRL ends it before the env's
  last step, and the episode is closed unscored.
* **One deploy per rollout.** Every rollout deploys its envs fresh, and the deploy counts in its time.
* **A server killed hard leaves its envs running.** Stop it with SIGTERM, which tears its episodes
  down.
* **Local Docker runs out of networks past about 30 episodes.** With the `local` sandbox, each
  episode of an env that runs in a container gets a Docker network of its own, and Docker's default
  address pools hold about 30. Add wider pools to Docker's `daemon.json`, and restart Docker before
  starting the server:

  ```json title="/etc/docker/daemon.json"
  { "default-address-pools": [{ "base": "10.200.0.0/14", "size": 24 }] }
  ```