SkyRL

How a rollout runsCopy Anchor Link

SkyRL plays multi-turn rollouts in its generator: for each row, it builds the env the row's env_class names, generates on its own vLLM, and hands each reply to the env's step, whose answer joins the context until the env says the episode is done. AgentEnv's env plays each row's task as one episode on a bundle's OpenEnv server. Four pieces connect SkyRL to a bundle:

SkyRL's tokenization, turn cap and training stay SkyRL's own, and it calls no reward function: the task's verifier scores each episode. SkyRL's generator shows the model no tools and reads no tool calls, so the writer puts the tools in each row's system prompt and the env reads the calls.

InstallCopy Anchor Link

SkyRL runs on Linux with NVIDIA GPUs. This page's runs used one L40S, the trainer and vLLM sharing it. Run SkyRL from its repository in its CUDA 13 image, and install the framework into its environment:

docker run -d --name skyrl --gpus all --network host --ipc host --shm-size 32g \
  -v "$PWD:$PWD" -w "$PWD" novaskyai/skyrl-train-ray-2.57.0-py3.12-cu13.0 sleep infinity
docker exec skyrl bash -c "git clone https://github.com/NovaSky-AI/SkyRL && cd SkyRL && uv sync --extra fsdp"
docker exec skyrl bash -c "cd SkyRL && uv pip install --python .venv/bin/python 'agentenv-framework[skyrl]>=0.9.1404'"

Run SkyRL's commands with docker exec skyrl SkyRL/.venv/bin/python .... The bundle's server runs where its envs deploy, with Docker running, in an environment of its own:

pip install 'agentenv-framework[openenv]>=0.9.1404'

Train on GridSimCopy Anchor Link

GridSim is the bundle the TRL page trains on. Each task runs one grid battery for one day in the ERCOT day-ahead market: the model reads the day's 24 hourly prices and submits a schedule, one integer an hour, to charge, sell or idle. The env grades the schedule against the day's exact optimum: 0 for nothing useful, 0.2 for idling all day, 1 for the optimum. The bundle has 2,000 train days and 200 eval days.

Serve the bundleCopy Anchor Link

hf download earakely-scale/GridSimEnv-AgentEnv --repo-type dataset --include "bundles/gridsim-v1/*" --local-dir .

import uvicorn

from agent_env.openenv import build_app

app = build_app("bundles/gridsim-v1", max_concurrent_envs=64)
uvicorn.run(app, host="127.0.0.1", port=8200)

max_concurrent_envs covers a step's 32 rollouts; an eval's 200 rollouts play 64 at a time.

Write the rowsCopy Anchor Link

python -m agent_env.training.skyrl http://127.0.0.1:8200 data \
  --tokenizer Qwen/Qwen3-4B --max-prompt-length 3072 --chat-template-kwargs '{"enable_thinking": false}'

wrote all.parquet, eval.parquet, train.parquet to data
the longest prompt is 1169 tokens with its tools

The command plays one task of each toolset to list the tools its rows offer, so it deploys an env once. Each row holds the task's prompt as chat messages, env_class: agentenv, the split, index and task id the env plays with the row's tools in episode, and the server's settings in server.

SkyRL's generator passes the chat template no tools, so --tokenizer puts each row's tools in its system prompt as the model's chat template renders them with tools=, and checks the prompt renders the same text either way: the model sees its tools as it would anywhere else. A template that renders tools outside the system turn is refused. --chat-template-kwargs is SkyRL's generator.chat_template_kwargs. A row over --max-prompt-length is refused before anything is written, counted both as SkyRL's prompt filter counts it, without the template settings, and as its generator renders it, with them; the filter would drop it without saying.

--tool-format names the format the template teaches the model to call tools in, which the env reads: hermes, the default, for Qwen2.5 and Qwen3, and qwen3_coder for Qwen3.5 and Qwen3-Coder.

Warm start with SFTCopy Anchor Link

Played as it comes, Qwen3-4B idles: it scores 0.18 on the 200 eval days. Warm-start it with the TRL page's sft.py, in an environment with TRL, so it saves qwen3-4b-gridsim-sft. It took 26 minutes on one L40S.

Train with GRPOCopy Anchor Link

This run uses the other trainers' runs' settings: the same SFT warm start, LoRA r32/α64 on every linear layer, learning rate 2e-5 cosine with 3 warmup steps, 32 rollouts a step (4 days × 8) at temperature 1.1, thinking off, and at most four model turns a rollout. The parts that are about AgentEnv:

python -m agent_env.training.skyrl.train \
  data.train_data="['data/train.parquet']" data.val_data="['data/eval.parquet']" \
  environment.env_class=agentenv environment.skyrl_gym.max_env_workers=64 \
  generator.batched=false generator.max_turns=4 \
  generator.chat_template_kwargs.enable_thinking=false \
  generator.max_input_length=8192 trainer.max_prompt_length=3072 \
  ...

The whole command, as it ran, is skyrl_gridsim.sh. Run it in SkyRL's environment from the folder holding qwen3-4b-gridsim-sft and data/. It evaluates on the 200 eval days before the first step and every 20 steps, one rollout a day at the training temperature.

Step 4:
{'environment/avg_attempts': '1.0000',
 'environment/avg_failed': '0.0000',
 'environment/avg_score/grade': '0.4924',
 'environment/avg_tool_calls': '2.8125',
 'environment/avg_unparsed_calls': '0.1250',
 'generate/trajectory_time_env_mean': '20.7275',
 'generate/trajectory_time_llm_mean': '13.9976',
 'reward/avg_raw_reward': '0.4924',
 'timing/step': '76.3410',
 …}

reward/avg_raw_reward is the mean of the step's 32 rewards, here GridSim's one score, environment/avg_score/grade. The env's other metrics are under What SkyRL reports.

What it learnsCopy Anchor Link

All five start from the same SFT model and score it alike, within what one rollout a day can tell apart (about ±0.06). Through SkyRL the train reward rises from 0.55 over the first 10 steps to 0.66 over the last 20, and the eval reward from 0.54 to 0.61. GridSim's greedy rule scores 0.76 on these days. This is one run of each trainer, so read the gaps as pointers, not a ranking.

The 40 GRPO steps with the three evals took 62 minutes on one L40S: about 70 seconds a step, a little over half of it generation. No rollout failed or was played again.

Settings SkyRL needsCopy Anchor Link

Envs on another machineCopy Anchor Link

SkyRL's generator runs in a Ray task on any node of its cluster, and each env opens its session on the URL in its row, so the bundle's server must be one that node reaches. Give it a token, which every request and WebSocket but /health must then carry:

import os

import uvicorn

from agent_env.openenv import build_app

app = build_app("bundles/gridsim-v1", max_concurrent_envs=64, token=os.environ["AGENTENV_TOKEN"])
uvicorn.run(app, host="0.0.0.0", port=8200)

python -m agent_env.training.skyrl http://envs.internal:8200 data --token-var AGENTENV_TOKEN ...

--token-var names the environment variable that holds the token: the writer reads it, and the rows record its name, never the token. agent_env.training.skyrl.train reads it where it starts and hands it to the Ray task that generates, on whichever node it runs. Without a token, anyone who reaches the server deploys and drives its envs.

The server speaks plain HTTP and WebSockets, so the token crosses the network unencrypted. Keep the server on a private network that only SkyRL's nodes reach, or put a TLS proxy in front of it and give the writer its https:// URL, which the env's sessions open as wss://.

Failed rolloutsCopy Anchor Link

A rollout the env fails, a deploy, a call past its deadline or a server that went away, is played again from the start on a new episode, a fresh sample of the same policy: up to attempts times, but none starts once reset_timeout has passed since the rollout began. SkyRL can't leave one rollout out of a step, and stops the step on an env's error, so AgentEnvGenerator leaves out the rollout's whole group instead, as SkyRL's own Harbor integration does with a failed trial: in training, it zeroes the group's loss masks and rewards, so the group is out of the gradient and its baseline. In an eval, the failed rollout scores 0.

Opening its session, a rollout waits for a server at its limit or one restarting, within reset_timeout, so a restart delays rollouts rather than failing them.

In two short runs on the GridSim rows, the env server was stopped while all 32 of a step's rollouts were mid-episode, and was back 40 to 50 seconds later:

A task whose episode ends unscored, or that offers a tool its row doesn't list, raises, and stops the step: no replay changes it.

What SkyRL reportsCopy Anchor Link

The env's metrics, whose means over each step's rollouts and each eval's SkyRL logs as environment/avg_<key> and eval/all/environment/avg_<key>. They're named as SkyRL names a mean, so where SkyRL joins groups' metrics, as its fully async trainer does, it averages them again rather than adding them up:

SkyRL's own generate/trajectory_time_llm_* and generate/trajectory_time_env_* split each rollout's time between the model and the env.

In the GridSim run a step took 70 seconds: 39 generating the 32 rollouts, 7 computing the old log-probs, 20 updating the actor and 3 loading the new weights into vLLM. A rollout spent 17 seconds in the env, its deploy included, and 11 in the model. An eval's 200 rollouts, 64 at a time, took 3 minutes.

The fully async trainerCopy Anchor Link

SkyRL's fully async trainer (trainer.fully_async.enabled=true) generates and trains at once, on GPUs of their own (trainer.placement.colocate_all=false): rollouts go on playing while the policy trains, on weights at most trainer.fully_async.max_staleness_steps behind. agent_env.training.skyrl.train picks it from that setting, and the env and generator need nothing else.

Both used SkyRL's own example's settings for the async trainer: max_staleness_steps=1, num_parallel_generation_workers=8 (train_batch_size × (max_staleness_steps + 1)), and its off-policy correction (trainer.algorithm.policy_loss_type=rollout_is with geometric sequence masking). The model was Qwen3-4B without the warm start, which idles toward 0.2 in either run; the comparison is of time, not of learning.

Multi-turn tasks and other tool-call formatsCopy Anchor Link

A task with several turns plays as one rollout. Each row offers end_turn with the task's tools: the model calls it with its answer, and its result is the next turn's prompt, which the env returns like any tool answer. When the model replies without calling a tool, its reply ends the turn it left open, and any turns after it end empty. Tell the model in the system prompt to end each turn with end_turn. A task that records several scores needs a reward that combines them, given to build_app, here their mean:

app = build_app("bundles/my-bundle", reward=lambda scores: sum(scores.values()) / len(scores),
                max_concurrent_envs=64)

For a model whose template teaches qwen3_coder, write the rows with --tool-format qwen3_coder; the env types each argument by its tool's JSON Schema. SkyRL loads Qwen3.5 as a vision-language model, so training it needs trainer.remove_microbatch_padding=false. A test bundle of four two-turn questions about an inbox, with a regex verifier a turn (answer-1 and answer-2), ran on Qwen3.5-2B for 5 GRPO steps of 32 rollouts, about 2 minutes a step:

<tool_call>
<function=search_emails>
<parameter=query>
invoice
</parameter>
</function>
</tool_call><|im_end|>
<|im_start|>user
<tool_response>
[{"id": "m-1", "from": "billing@acme-supplies.example", "subject": "Invoice INV-1042 for October", …}]
</tool_response><|im_end|>
<|im_start|>assistant
…
The invoice number in your inbox is INV-1042.<|im_end|>

The env read every call the model made, none unparsed. The first turn's score rose from 0.78 over step 1's rollouts to 0.97 over step 5's, and the eval from 0.375 to 0.5, all of it on the first turn: like the rollout above, the model answered the first question in a reply rather than with end_turn, so it never saw the second, as it did under verl. The reward alone doesn't show that; environment/avg_score/answer-2, 0 at every step, does.

What's supportedCopy Anchor Link

LimitsCopy Anchor Link

• A failed rollout leaves its group out. SkyRL has no way to leave out one rollout, so the rollouts of its group are left out with it.

• A rollout that outgrows generator.max_input_length scores 0. SkyRL ends it before the env's last step, and the episode is closed unscored.

• One deploy per rollout. Every rollout deploys its envs fresh, and the deploy counts in its time.

• A server killed hard leaves its envs running. Stop it with SIGTERM, which tears its episodes down.

• Local Docker runs out of networks past about 30 episodes. With the local sandbox, each episode of an env that runs in a container gets a Docker network of its own, and Docker's default address pools hold about 30. Add wider pools to Docker's daemon.json, and restart Docker before starting the server:

  Copy Text

  { "default-address-pools": [{ "base": "10.200.0.0/14", "size": 24 }] }