# prime-rl (https://www.agentenvframework.com/docs/training/prime-rl)

> Train using Prime Intellect's prime-rl

## How a rollout runs

1. prime-rl’s orchestrator hands a row of the split’s JSONL file to verifiers’ NeMo Gym taskset, which seeds a session on the resources server with it.
2. The server starts the row’s task as an episode: it deploys the env, loads the task, and opens the agent’s turn, while your model writes its first reply.
3. Your model plays the turn: the taskset posts each native tool call to the tool’s route on the server, which runs it on the env through the episode.
4. Your model ends its turn with a reply that calls no tool, or the turn cap ends it, and the taskset posts the rollout to `/verify`. The task’s verifier scores the episode.
5. `/verify` answers with the score as the reward, and with an error for a rollout the environment failed, which prime-rl leaves out of the update; the episode is torn down.

Parts of the scene:

- **prime-rl and verifiers**: prime-rl’s orchestrator plays each row through verifiers’ built-in NeMo Gym taskset with the `null` harness, sampling your model from prime-rl’s own vLLM. prime-rl owns the model, the generation and the update; the environment owns the reward.
- **A dataset row**: A line of `data/<split>.jsonl`, the rows NeMo RL trains on too: the task’s prompt and tools as `responses_create_params`, and the `split`, `index` and `task_id` the server plays. The prompt is yours to change; the task stays the same.
- **The resources server**: `python -m agent_env.training.nemo_gym serve` runs the server on its own, at a URL the taskset takes as `resources_url`. `/seed_session` starts a row’s task, a route per tool calls it, and `/verify` ends the turn and answers with the reward.
- **The session**: One session per rollout, by cookie, holding one session on the bundle’s server, so one episode. prime-rl keeps rollouts in flight across steps, so cap them with `max_inflight` at what the bundle’s server runs at once.
- **The OpenEnv server**: `agent_env.openenv.build_app(bundle)` serves every task of a bundle as an OpenEnv env, the split `all` plus one per eval. Serve it on a port other than 8000, which prime-rl’s inference takes.
- **The env**: The task’s env, deployed fresh for this episode in a sandbox, here a container on your Docker. Its tools are what the model calls.
- **The agent’s turn**: The task prompts one agent, and your model plays it from outside: the episode deploys a proxy in the agent’s place and hands the turn’s prompt and tools to the server.
- **The verifier**: The task’s own verifier step scores the episode once the turn ends, the same score an evaluation run records. With several scores, `reward=` picks the one to train on.
- **The reward**: `/verify` returns the score as `reward`. verifiers reads `mask_sample` only as a metric, so `serve --masked-as-error` answers a rollout the environment failed with an error, which prime-rl leaves out of the loss and the group’s baseline.

prime-rl trains on environments through [verifiers](https://github.com/PrimeIntellect-ai/verifiers),
whose built-in NeMo Gym taskset plays any NeMo Gym resources server: it seeds a session with a
dataset row, calls the row's tools, and asks the server to verify the rollout for a reward. That is
the server the [NeMo RL page](https://www.agentenvframework.com/docs/training/nemo-rl.md) trains through, run here on its own. Four
pieces connect prime-rl to a bundle:

| Piece                                         | What it does                                                                                                                                  |
| --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `agent_env.openenv.build_app(bundle)`         | Serves every task of a bundle as an [OpenEnv](https://www.agentenvframework.com/docs/openenv.md) env: the split `all`, plus one split per eval.                                   |
| `python -m agent_env.training.nemo_gym`       | Writes the bundle's resources server into a folder: the server, its config, and a JSONL file of rows per split.                               |
| `python -m agent_env.training.nemo_gym serve` | Runs that resources server on its own, at a URL, without NeMo Gym's other servers.                                                            |
| verifiers' NeMo Gym taskset                   | `env.taskset.id = "nemo-gym"` with the `null` harness: plays each row with your model, served from prime-rl's vLLM, against the server's URL. |

## Install

prime-rl runs on Linux with at least two NVIDIA GPUs, one to train and one to serve the model. Clone
v0.10.0 and the submodules it needs, which include the verifiers commit it trains with:

```bash title="Terminal"
git clone --branch v0.10.0 https://github.com/PrimeIntellect-ai/prime-rl.git
cd prime-rl
git submodule update --init -- deps/verifiers deps/renderers deps/prime-envs deps/pydantic-config
uv sync --all-extras
```

The submodules are listed by SSH URL. On a machine without a GitHub SSH key, run
`git config --global url."https://github.com/".insteadOf "git@github.com:"` first.

Then, in an environment of its own on the machine where the envs will deploy, with Docker running,
install the framework to serve the bundle and the resources server:

```bash title="Terminal"
pip install 'agentenv-framework[nemo-gym]>=0.9.1399'
```

`serve` imports NeMo Gym, which needs Python 3.13.14 or newer. Writing the server takes only
OpenEnv, from Python 3.11.

> [!WARNING]
> **On AWS's Deep Learning AMI**
>
> The page's run used the Deep Learning Base OSS NVIDIA Driver AMI (Ubuntu 24.04) on four L40S. Two of
> its defaults get in prime-rl's way:
>
> * NVIDIA DCGM listens on port 5555, prime-rl's default port between the orchestrator and the
>   trainer. The config below moves it with `[rollout_transport] port = 5565`.
> * FlashInfer compiles its sampling kernel on first use, with the `ninja` in prime-rl's venv. Run
>   prime-rl through `uv run`, which puts that venv on `PATH`; the run also set
>   `CUDA_HOME=/usr/local/cuda-13.0`, the toolkit that matches its torch build.

## Train on GridSim

GridSim is the bundle the [TRL page](https://www.agentenvframework.com/docs/training/trl.md#train-on-gridsim) trains on. Each task runs
one grid battery for one day in the ERCOT day-ahead market: the model reads the day's 24 hourly
prices and submits a schedule, one integer an hour, to charge, sell or idle. The env grades the
schedule against the day's exact optimum: 0 for nothing useful, 0.2 for idling all day, 1 for the
optimum. The bundle has 2,000 train days and 200 eval days.

### Serve the bundle

Download the bundle, then serve it on the machine prime-rl runs on:

```bash title="Terminal"
hf download earakely-scale/GridSimEnv-AgentEnv --repo-type dataset --include "bundles/gridsim-v1/*" --local-dir .
```

```python title="serve.py"
import uvicorn

from agent_env.openenv import build_app

app = build_app("bundles/gridsim-v1", max_concurrent_envs=96)
uvicorn.run(app, host="127.0.0.1", port=8200)
```

Keep it off port 8000, where prime-rl's inference router listens.

### Write the resources server

From the `prime-rl` folder, write the server into a folder of its own:

```bash title="Terminal"
python -m agent_env.training.nemo_gym http://127.0.0.1:8200 gym-root
```

```text title="Output"
wrote gym-root/resources_servers/gridsim_v1
```

The command plays one task of each toolset to list the tools its rows offer, so it deploys an env
once. It writes the same server, config and rows as on the [NeMo RL page](https://www.agentenvframework.com/docs/training/nemo-rl.md#write-the-resources-server):
`data/train.jsonl` with 2,000 rows and `data/eval.jsonl` with 200.

### Serve the resources server

```bash title="Terminal"
python -m agent_env.training.nemo_gym serve gym-root/resources_servers/gridsim_v1 --port 9100 \
  --masked-as-error --idle-timeout 120
```

`serve` runs the folder's server on its own, on loopback by default, since it checks no token.
`--masked-as-error` and `--idle-timeout` are for verifiers, which reads neither `mask_sample` nor
closes the sessions it abandons: see [failed and abandoned rollouts](#failed-and-abandoned-rollouts).

### Warm start with SFT

Played as it comes, Qwen3-4B idles: it scores 0.18 on the 200 eval days. Warm-start it with the
TRL page's [`sft.py`](https://www.agentenvframework.com/docs/training/trl.md#warm-start-with-sft), in an environment with TRL, run from
the `prime-rl` folder so it saves `qwen3-4b-gridsim-sft` where the config below looks. It took 25
minutes on one L40S.

### Train with GRPO

prime-rl takes a TOML config. This run uses the TRL and NeMo RL runs' settings: the same 32
rollouts a step, learning rate, schedule and LoRA, and NeMo RL's cap of four model turns a rollout.
The parts that matter:

```toml title="prime_rl_gridsim.toml (excerpt)"
max_steps = 40

[deployment]
num_train_gpus = 1
num_infer_gpus = 3

[model]
name = "qwen3-4b-gridsim-sft"

[trainer.optim]
lr = 2e-5

[trainer.model.lora]
rank = 32
alpha = 64.0

[orchestrator]
batch_size = 32  # 4 days, 8 rollouts each

[orchestrator.concurrency]
max_inflight = 64

[orchestrator.renderer]
name = "qwen3"
enable_thinking = false

[orchestrator.train]
group_size = 8

[orchestrator.train.sampling]
temperature = 1.1
max_completion_tokens = 1024

[[orchestrator.train.source]]
name = "gridsim"
env.taskset.id = "nemo-gym"
env.taskset.dataset = "gym-root/resources_servers/gridsim_v1/data/train.jsonl"
env.taskset.task.resources_url = "http://127.0.0.1:9100"
env.taskset.task.request_timeout = 600
env.agent.harness.id = "null"
env.agent.max_turns = 4
env.agent.runtime.type = "subprocess"

[orchestrator.eval]
interval = 20

[inference.vllm]
tool_call_parser = "hermes"
```

The whole file, as it ran with its paths made relative to the `prime-rl` folder, is
[`prime_rl_gridsim.toml`](/training/prime_rl_gridsim.toml). It evaluates on the 200 eval days before
the first step and every 20 steps, one rollout a day at the training temperature. Save it in the
`prime-rl` folder and run:

```bash title="Terminal"
uv run rl @ prime_rl_gridsim.toml
```

prime-rl starts its vLLM on three GPUs and the trainer on the fourth, and an env server per source
that runs verifiers' taskset against the resources server. prime-rl renders Qwen3's chat template
itself, so thinking is turned off in `[orchestrator.renderer]`; the `hermes` parser turns Qwen3's tool
calls into the native calls the taskset makes.

```text title="Output (step 12 of a GridSim run with this config, trimmed)"
SUCCESS Step 12 |  1m 23s | Reward 0.7508 | Trainable 32/32 (100.0%) | Turns 3.9 | Max Off-Policy 4 | Error 0.0% | …
```

`Reward` is the mean of the step's 32 rewards. Most rollouts reach the four-turn cap, which prime-rl
counts as truncation; their reward is the episode's score all the same.

### What it learns

| Qwen3-4B | eval reward (200 days) |
|---|---|
| after SFT (GRPO step 0) | 0.565 |
| GRPO step 20 | 0.580 |
| GRPO step 40 | 0.566 |

Mean train reward of each GRPO step's 32 rollouts, by step: 1: 0.776, 2: 0.571, 3: 0.810, 4: 0.351, 5: 0.821, 6: 0.555, 7: 0.686, 8: 0.340, 9: 0.829, 10: 0.522, 11: 0.745, 12: 0.651, 13: 0.390, 14: 0.720, 15: 0.783, 16: 0.661, 17: 0.604, 18: 0.612, 19: 0.903, 20: 0.822, 21: 0.646, 22: 0.717, 23: 0.849, 24: 0.668, 25: 0.725, 26: 0.782, 27: 0.617, 28: 0.884, 29: 0.802, 30: 0.742, 31: 0.545, 32: 0.595, 33: 0.634, 34: 0.765, 35: 0.827, 36: 0.653, 37: 0.857, 38: 0.631, 39: 0.920, 40: 0.660.

| Qwen3-4B, eval reward on the 200 eval days | prime-rl | NeMo RL | TRL  |
| ------------------------------------------ | -------- | ------- | ---- |
| After SFT (GRPO step 0)                    | 0.56     | 0.55    | 0.57 |
| After 20 GRPO steps                        | 0.58     | 0.68    | 0.56 |
| After 40 GRPO steps                        | 0.57     | 0.67    | 0.61 |

All three start from the same SFT model and score it alike, within what one rollout a day can tell
apart (about ±0.06). Through prime-rl the train reward rises, from 0.63 over the first 10 steps to
0.71 to 0.74 over the last 20, but the eval reward stays where it started. GridSim's greedy rule
scores 0.76 on these days.

The runs share the SFT model, rows, LoRA, learning rate, schedule, rollouts a step and turn cap. The
known differences are the trainers' own: prime-rl's GRPO loss and its asynchronous rollouts, a few
steps off-policy at times. This is one run of each, so read the gap as a pointer, not a ranking of
the trainers.

The GRPO run took 46 minutes on four L40S, an AWS g6e.12xlarge that also served the envs: about a
minute a step, and a few minutes for each of the three evals.

## Settings prime-rl needs

Each of these was a failure on the way to the run above:

| Setting                                   | Value here                         | Without it                                                                                                                                                                                                            |
| ----------------------------------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| The bundle's port                         | 8200                               | prime-rl's inference router can't bind 8000, its inference pool shuts down, and the run waits forever.                                                                                                                |
| `env.agent.max_turns`                     | 4                                  | verifiers has no turn cap by default: a rollout ran 165 turns, each one a model call.                                                                                                                                 |
| `[orchestrator.concurrency] max_inflight` | 64                                 | prime-rl sizes its rollouts in flight to its own GPUs, not to the bundle's server: it reached about 700 against 96 episodes, and rollouts timed out waiting for a session. Keep it at or below `max_concurrent_envs`. |
| `env.taskset.task.request_timeout`        | 600                                | verifiers waits 60 seconds per request to the server. Under load, `/verify` took longer, and 12 to 20% of eval rollouts failed.                                                                                       |
| `[orchestrator.renderer]`                 | `qwen3`, `enable_thinking = false` | Qwen3-4B is a thinking model, and prime-rl, not vLLM, renders its chat template.                                                                                                                                      |
| `serve --masked-as-error`                 | on                                 | A rollout the environment failed trains as a reward of 0.                                                                                                                                                             |
| `serve --idle-timeout`                    | 120 s                              | Rollouts prime-rl cancels hold their episodes for 600 seconds; the final eval found the bundle's server full.                                                                                                         |

## Failed and abandoned rollouts

A rollout the environment failed is masked: a deploy that failed, a call that missed its deadline, a
server that went away. NeMo RL reads `mask_sample` from `/verify` and leaves the rollout out of the
loss, but verifiers records it only as a metric and trains on the reward of 0. With
`--masked-as-error`, `/verify` answers a masked rollout with HTTP 503 and the reason instead.
verifiers then fails the rollout, and prime-rl leaves failed rollouts out of the loss and out of
their group's baseline. A change proposed to verifiers
([PrimeIntellect-ai/verifiers#3016](https://github.com/PrimeIntellect-ai/verifiers/pull/3016)) would
drop masked rollouts on its side. The flag is off by default, since NeMo RL reads `mask_sample` from
a normal answer.

verifiers never calls `/close_session`, so a rollout prime-rl cancels, as it does with the rollouts
still in flight when training ends, leaves its episode running until the server's idle timeout ends
it. `--idle-timeout 120` reclaims it after two minutes instead of ten; keep it above the longest
gap between your model's calls in one rollout. The run above masked none of its 1,978 rollouts and
reclaimed 55 abandoned ones.

When the bundle's server is too busy to answer, a rollout's session tries again for up to
`reset_timeout` (600 seconds), as it does when the server is at its limit.

## Check without training

verifiers' `vf-eval` plays the same rows against the server without prime-rl, with any
OpenAI-compatible endpoint that supports tool calling. For the SFT model on vLLM, serve it with
`--enable-auto-tool-choice --tool-call-parser hermes`:

```toml title="eval.toml"
model = "sft"

[client]
base_url = "http://127.0.0.1:8100/v1"
api_key_var = "VLLM_API_KEY"

[sampling]
temperature = 1.1
max_tokens = 1024
chat_template_kwargs = { enable_thinking = false }

[env.taskset]
id = "nemo-gym"
dataset = "gym-root/resources_servers/gridsim_v1/data/eval.jsonl"

[env.taskset.task]
resources_url = "http://127.0.0.1:9100"

[env.agent]
max_turns = 4

[env.agent.harness]
id = "null"

[env.agent.runtime]
type = "subprocess"
```

```bash title="Terminal, in the prime-rl folder"
VLLM_API_KEY=unused uv run vf-eval @ eval.toml -n 8 -r 1 --no-push
```

Run on 8 eval days without the turn cap, the SFT model scored 0.77, every rollout through the
server's routes. Played this way, verifiers' rewards match what the Episode API scores for the
same schedules.

## What's supported

|                               |                                                                                                                                              |
| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| prime-rl                      | v0.10.0 with the verifiers commit it pins, GRPO with LoRA, trained on GPUs.                                                                  |
| verifiers                     | 0.4.0 and main. A CI job plays rows through `serve` with `vf-eval` on both, every week.                                                      |
| Multi-turn tasks              | `end_turn` hands the model each next prompt.                                                                                                 |
| Bundles with several toolsets | Each row lists its own task's tools.                                                                                                         |
| Sandboxes                     | `build_app(..., sandbox=)` deploys every episode where `agent-env run --sandbox` would, from your Docker to [any provider](https://www.agentenvframework.com/docs/sandboxes.md). |

## Limits

* **You run the server.** verifiers' NeMo Gym taskset can start only servers from NeMo Gym's own
  package, so `serve` runs ours and the taskset takes its URL.
* **No Environments Hub or Hosted Training yet.** Both run environments from Prime Intellect's cloud,
  which would need the server reachable from there, and it checks no token.
* **One process per bundle.** The resources server holds every rollout's session in one worker. For
  more rollouts than one process serves, write the bundle as several servers (`--name`), split the
  rows between them, and serve each.
* **One deploy per rollout.** Every rollout deploys its envs fresh, and the deploy counts in its time.