prime-rl

How a rollout runsCopy Anchor Link

prime-rl trains on environments through verifiers, whose built-in NeMo Gym taskset plays any NeMo Gym resources server: it seeds a session with a dataset row, calls the row's tools, and asks the server to verify the rollout for a reward. That is the server the NeMo RL page trains through, run here on its own. Four pieces connect prime-rl to a bundle:

InstallCopy Anchor Link

prime-rl runs on Linux with at least two NVIDIA GPUs, one to train and one to serve the model. Clone v0.10.0 and the submodules it needs, which include the verifiers commit it trains with:

git clone --branch v0.10.0 https://github.com/PrimeIntellect-ai/prime-rl.git
cd prime-rl
git submodule update --init -- deps/verifiers deps/renderers deps/prime-envs deps/pydantic-config
uv sync --all-extras

The submodules are listed by SSH URL. On a machine without a GitHub SSH key, run git config --global url."https://github.com/".insteadOf "git@github.com:" first.

Then, in an environment of its own on the machine where the envs will deploy, with Docker running, install the framework to serve the bundle and the resources server:

pip install 'agentenv-framework[nemo-gym]>=0.9.1399'

serve imports NeMo Gym, which needs Python 3.13.14 or newer. Writing the server takes only OpenEnv, from Python 3.11.

Train on GridSimCopy Anchor Link

GridSim is the bundle the TRL page trains on. Each task runs one grid battery for one day in the ERCOT day-ahead market: the model reads the day's 24 hourly prices and submits a schedule, one integer an hour, to charge, sell or idle. The env grades the schedule against the day's exact optimum: 0 for nothing useful, 0.2 for idling all day, 1 for the optimum. The bundle has 2,000 train days and 200 eval days.

Serve the bundleCopy Anchor Link

Download the bundle, then serve it on the machine prime-rl runs on:

hf download earakely-scale/GridSimEnv-AgentEnv --repo-type dataset --include "bundles/gridsim-v1/*" --local-dir .

import uvicorn

from agent_env.openenv import build_app

app = build_app("bundles/gridsim-v1", max_concurrent_envs=96)
uvicorn.run(app, host="127.0.0.1", port=8200)

Keep it off port 8000, where prime-rl's inference router listens.

Write the resources serverCopy Anchor Link

From the prime-rl folder, write the server into a folder of its own:

python -m agent_env.training.nemo_gym http://127.0.0.1:8200 gym-root

wrote gym-root/resources_servers/gridsim_v1

The command plays one task of each toolset to list the tools its rows offer, so it deploys an env once. It writes the same server, config and rows as on the NeMo RL page: data/train.jsonl with 2,000 rows and data/eval.jsonl with 200.

Serve the resources serverCopy Anchor Link

python -m agent_env.training.nemo_gym serve gym-root/resources_servers/gridsim_v1 --port 9100 \
  --masked-as-error --idle-timeout 120

serve runs the folder's server on its own, on loopback by default, since it checks no token. --masked-as-error and --idle-timeout are for verifiers, which reads neither mask_sample nor closes the sessions it abandons: see failed and abandoned rollouts.

Warm start with SFTCopy Anchor Link

Played as it comes, Qwen3-4B idles: it scores 0.18 on the 200 eval days. Warm-start it with the TRL page's sft.py, in an environment with TRL, run from the prime-rl folder so it saves qwen3-4b-gridsim-sft where the config below looks. It took 25 minutes on one L40S.

Train with GRPOCopy Anchor Link

prime-rl takes a TOML config. This run uses the TRL and NeMo RL runs' settings: the same 32 rollouts a step, learning rate, schedule and LoRA, and NeMo RL's cap of four model turns a rollout. The parts that matter:

max_steps = 40

[deployment]
num_train_gpus = 1
num_infer_gpus = 3

[model]
name = "qwen3-4b-gridsim-sft"

[trainer.optim]
lr = 2e-5

[trainer.model.lora]
rank = 32
alpha = 64.0

[orchestrator]
batch_size = 32  # 4 days, 8 rollouts each

[orchestrator.concurrency]
max_inflight = 64

[orchestrator.renderer]
name = "qwen3"
enable_thinking = false

[orchestrator.train]
group_size = 8

[orchestrator.train.sampling]
temperature = 1.1
max_completion_tokens = 1024

[[orchestrator.train.source]]
name = "gridsim"
env.taskset.id = "nemo-gym"
env.taskset.dataset = "gym-root/resources_servers/gridsim_v1/data/train.jsonl"
env.taskset.task.resources_url = "http://127.0.0.1:9100"
env.taskset.task.request_timeout = 600
env.agent.harness.id = "null"
env.agent.max_turns = 4
env.agent.runtime.type = "subprocess"

[orchestrator.eval]
interval = 20

[inference.vllm]
tool_call_parser = "hermes"

The whole file, as it ran with its paths made relative to the prime-rl folder, is prime_rl_gridsim.toml. It evaluates on the 200 eval days before the first step and every 20 steps, one rollout a day at the training temperature. Save it in the prime-rl folder and run:

uv run rl @ prime_rl_gridsim.toml

prime-rl starts its vLLM on three GPUs and the trainer on the fourth, and an env server per source that runs verifiers' taskset against the resources server. prime-rl renders Qwen3's chat template itself, so thinking is turned off in [orchestrator.renderer]; the hermes parser turns Qwen3's tool calls into the native calls the taskset makes.

SUCCESS Step 12 |  1m 23s | Reward 0.7508 | Trainable 32/32 (100.0%) | Turns 3.9 | Max Off-Policy 4 | Error 0.0% | …

Reward is the mean of the step's 32 rewards. Most rollouts reach the four-turn cap, which prime-rl counts as truncation; their reward is the episode's score all the same.

What it learnsCopy Anchor Link

All three start from the same SFT model and score it alike, within what one rollout a day can tell apart (about ±0.06). Through prime-rl the train reward rises, from 0.63 over the first 10 steps to 0.71 to 0.74 over the last 20, but the eval reward stays where it started. GridSim's greedy rule scores 0.76 on these days.

The runs share the SFT model, rows, LoRA, learning rate, schedule, rollouts a step and turn cap. The known differences are the trainers' own: prime-rl's GRPO loss and its asynchronous rollouts, a few steps off-policy at times. This is one run of each, so read the gap as a pointer, not a ranking of the trainers.

The GRPO run took 46 minutes on four L40S, an AWS g6e.12xlarge that also served the envs: about a minute a step, and a few minutes for each of the three evals.

Settings prime-rl needsCopy Anchor Link

Each of these was a failure on the way to the run above:

Failed and abandoned rolloutsCopy Anchor Link

A rollout the environment failed is masked: a deploy that failed, a call that missed its deadline, a server that went away. NeMo RL reads mask_sample from /verify and leaves the rollout out of the loss, but verifiers records it only as a metric and trains on the reward of 0. With --masked-as-error, /verify answers a masked rollout with HTTP 503 and the reason instead. verifiers then fails the rollout, and prime-rl leaves failed rollouts out of the loss and out of their group's baseline. A change proposed to verifiers (PrimeIntellect-ai/verifiers#3016) would drop masked rollouts on its side. The flag is off by default, since NeMo RL reads mask_sample from a normal answer.

verifiers never calls /close_session, so a rollout prime-rl cancels, as it does with the rollouts still in flight when training ends, leaves its episode running until the server's idle timeout ends it. --idle-timeout 120 reclaims it after two minutes instead of ten; keep it above the longest gap between your model's calls in one rollout. The run above masked none of its 1,978 rollouts and reclaimed 55 abandoned ones.

When the bundle's server is too busy to answer, a rollout's session tries again for up to reset_timeout (600 seconds), as it does when the server is at its limit.

Check without trainingCopy Anchor Link

verifiers' vf-eval plays the same rows against the server without prime-rl, with any OpenAI-compatible endpoint that supports tool calling. For the SFT model on vLLM, serve it with --enable-auto-tool-choice --tool-call-parser hermes:

model = "sft"

[client]
base_url = "http://127.0.0.1:8100/v1"
api_key_var = "VLLM_API_KEY"

[sampling]
temperature = 1.1
max_tokens = 1024
chat_template_kwargs = { enable_thinking = false }

[env.taskset]
id = "nemo-gym"
dataset = "gym-root/resources_servers/gridsim_v1/data/eval.jsonl"

[env.taskset.task]
resources_url = "http://127.0.0.1:9100"

[env.agent]
max_turns = 4

[env.agent.harness]
id = "null"

[env.agent.runtime]
type = "subprocess"

VLLM_API_KEY=unused uv run vf-eval @ eval.toml -n 8 -r 1 --no-push

Run on 8 eval days without the turn cap, the SFT model scored 0.77, every rollout through the server's routes. Played this way, verifiers' rewards match what the Episode API scores for the same schedules.

What's supportedCopy Anchor Link

LimitsCopy Anchor Link

• You run the server. verifiers' NeMo Gym taskset can start only servers from NeMo Gym's own package, so serve runs ours and the taskset takes its URL.
• No Environments Hub or Hosted Training yet. Both run environments from Prime Intellect's cloud, which would need the server reachable from there, and it checks no token.
• One process per bundle. The resources server holds every rollout's session in one worker. For more rollouts than one process serves, write the bundle as several servers (--name), split the rows between them, and serve each.
• One deploy per rollout. Every rollout deploys its envs fresh, and the deploy counts in its time.