Unsloth

How a rollout runsCopy Anchor Link

Unsloth's GRPOTrainer is TRL's, rewritten when unsloth is imported first. With fast_inference=True it generates on a vLLM in its own process that shares the model's weights, so GRPO on a 4B model fits one GPU, in 4-bit a 24 GB one. It takes the TRL helper's dataset and environments as they are:

Unsloth's tokenization, generation and training stay its own, and it calls no reward function of yours: the task's verifier scores each episode.

InstallCopy Anchor Link

You need Linux with an NVIDIA GPU, Python 3.11 or newer, and Docker running:

pip install 'agentenv-framework[trl]>=0.9.1407' 'unsloth==2026.10.11' 'unsloth_zoo==2026.10.11' \
  'trl==1.15.0' 'transformers==5.17.0' 'vllm==0.29.0' 'vllm-bnb-plugin==0.0.3'

Train on GridSimCopy Anchor Link

GridSim is the bundle the TRL page trains on. Each task runs one grid battery for one day in the ERCOT day-ahead market: the model reads the day's 24 hourly prices and submits a schedule, one integer an hour, to charge, sell or idle. The env grades the schedule against the day's exact optimum: 0 for nothing useful, 0.2 for idling all day, 1 for the optimum. The bundle has 2,000 train days and 200 eval days.

hf download earakely-scale/GridSimEnv-AgentEnv --repo-type dataset --include "bundles/gridsim-v1/*" --local-dir .

Warm start with SFTCopy Anchor Link

Played as it comes, Qwen3-4B idles: it scores 0.18 on the 200 eval days. Warm-start it with the TRL page's sft.py, in an environment set up as that page's Install has it, so it saves qwen3-4b-gridsim-sft. It took 25 minutes on one L40S.

Train with GRPOCopy Anchor Link

This run uses the other trainers' runs' settings: the same SFT warm start, LoRA r32/α64 on every linear layer, learning rate 2e-5 cosine with 3 warmup steps, 32 rollouts a step (4 days × 8) at temperature 1.1, thinking off, and at most two tool-calling rounds a rollout. It sets back the defaults Unsloth changes (under Settings Unsloth changes), so its row compares with the TRL run's:

from unsloth import FastLanguageModel  # first, so Unsloth rewrites TRL's trainers

from trl import GRPOConfig, GRPOTrainer

from agent_env.training.trl import AgentEnv

env = AgentEnv.local("bundles/gridsim-v1", split="train", max_concurrent_envs=32)
model, tokenizer = FastLanguageModel.from_pretrained(
    "qwen3-4b-gridsim-sft", max_seq_length=4096, load_in_4bit=False, fast_inference=True,
    max_lora_rank=32, gpu_memory_utilization=0.5)
model = FastLanguageModel.get_peft_model(
    model, r=32, lora_alpha=64,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    use_gradient_checkpointing="unsloth", random_state=0)
trainer = GRPOTrainer(
    model=model,
    processing_class=tokenizer,
    args=GRPOConfig(
        output_dir="out",
        max_steps=40,
        per_device_train_batch_size=8,
        gradient_accumulation_steps=4,  # 32 rollouts a step: 4 days, 8 rollouts each
        num_generations=8,
        max_completion_length=1024,
        max_tool_calling_iterations=2,
        chat_template_kwargs={"enable_thinking": False},
        temperature=1.1,
        learning_rate=2e-5,
        lr_scheduler_type="cosine",
        warmup_steps=3,
        loss_type="dr_grpo",
        scale_rewards=None,  # Unsloth turns reward scaling off for dr_grpo unless it's None
        beta=0.0,
        epsilon_high=0.28,
        optim="adamw_torch",  # AdamW, as TRL's adamw_torch_fused; Unsloth's is adamw_8bit
        weight_decay=0.0,  # TRL's default; Unsloth's is 0.001
        use_vllm=True,
        bf16=True,
        logging_steps=1,
        eval_strategy="steps",
        eval_steps=20,
        eval_on_start=True,
        num_generations_eval=1,
        per_device_eval_batch_size=32,
        save_strategy="no",
        seed=0,
        report_to="none",
    ),
    train_dataset=env.dataset,
    eval_dataset=env.dataset_for("eval"),
    environment_factory=env.factory,
)
trainer.train()

There are no reward_funcs: each environment owns its reward through get_reward(), which the trainer reads once the rollout ends. TRL's vllm_* settings don't apply: Unsloth's vLLM takes its memory and context from from_pretrained.

{"loss": 0.0523, "grad_norm": 0.0498, "reward": 0.6434, "reward_std": 0.3533, "rewards/gridsim-v1/mean": 0.6434,
 "tools/call_frequency": 2.0, "tools/failure_frequency": 0.0, "completions/mean_length": 681.2, "step_time": 101.8}

The reward column is named after the environment class, here the bundle's name. The server needs a session for each rollout in flight: 32 a step, and 32 an eval batch.

What it learnsCopy Anchor Link

All six start from the same SFT model and score it alike, within what one rollout a day can tell apart (about ±0.06). Through Unsloth, as through TRL, the train reward stays level, at 0.66 over the first 10 steps and over the last 20, and the eval ends at 0.59. GridSim's greedy rule scores 0.76 on these days. This is one run of each trainer, so read the gaps as pointers, not a ranking.

The 40 GRPO steps with the three evals took 103 minutes on one L40S with 16 vCPUs, an AWS g6e.4xlarge: about 104 seconds a step, and 9 to 11 minutes an eval. No rollout failed. The TRL run's steps took about 190 seconds on a g6e.xlarge with 4 vCPUs, most of it deploying the envs, so the two times compare machines as much as trainers.

Settings Unsloth changesCopy Anchor Link

Unsloth's GRPOConfig gives these arguments other defaults when you leave them unset, and changes one you set:

The warmup counts from 0, so the first step trains at learning rate 0 unless warmup_steps=0.

On a 24 GB GPUCopy Anchor Link

The same SFT model, loaded in 4-bit, trained with the run's settings on one L4 with 24 GB, an AWS g6.4xlarge, using about 22 GB of it. Only the loading changes:

model, tokenizer = FastLanguageModel.from_pretrained(
    "qwen3-4b-gridsim-sft", max_seq_length=4096, load_in_4bit=True, fast_inference=True,
    max_lora_rank=32, gpu_memory_utilization=0.5)

The 4-bit model starts a little below the 16-bit one, and its 10 steps leave it within what one rollout a day can tell apart. A 40-step run with three evals would take about four and a half hours on the L4.

Serve the bundle on its own hostCopy Anchor Link

To keep the envs off the GPU machine, serve the bundle where they can deploy, and point the trainer at it. The rest of train.py stays the same:

import uvicorn

from agent_env.openenv import build_app

app = build_app("bundles/gridsim-v1", max_concurrent_envs=48)
uvicorn.run(app, host="0.0.0.0", port=8000)

env = AgentEnv("http://env-host:8000", split="train")

Each environment holds one session on the server, so max_concurrent_envs must cover every rollout in flight, here the 32 of a step and the 32 of an eval batch. build_app(..., sandbox="modal") deploys each episode's envs on another sandbox provider instead of the host's Docker, for a GPU machine without Docker, such as a notebook's.

Failed rolloutsCopy Anchor Link

get_reward() returns None when an episode ended in an error: a deploy that failed, a call past its deadline, or a server that went away. Unsloth's trainer, like TRL's, leaves that rollout out of the update and of its group's baseline, and logs it:

WARNING agent_env.training.trl: rollout left unscored: end_turn: ConnectionError: Failed to connect to ws://127.0.0.1:8200/ws: [Errno 111] Connect call failed ('127.0.0.1', 8200)

A rollout isn't played again: the step trains on the rollouts that scored. An environment holds its session from one rollout to the next, and when the server restarts, the next rollout waits for it to come back, within reset_timeout, rather than failing. In two short runs on the L4, with 16 rollouts a step, the env server was stopped 10 seconds into step 2 and was back 30 seconds later:

A step whose rollouts all ended unscored trains on nothing, and TRL warns that every reward function returned None.

Multi-turn tasksCopy Anchor Link

A task with several turns plays as one rollout: end_turn hands the model each next turn's prompt, as on the TRL page. The model's answer reaches the task only through end_turn(response=...), so a turn it answers in plain text ends with an empty reply, which a verifier judging the reply scores as such: tell the model in the system prompt to answer with end_turn. A task that records several scores needs a reward that combines them, here their mean:

env = AgentEnv.local("bundles/my-bundle", reward=lambda scores: sum(scores.values()) / len(scores))

A test bundle of four two-turn questions about an inbox, with a regex verifier a turn, trained on the L4 in 4-bit, Qwen3-4B, 5 GRPO steps of 32 rollouts at about 4 minutes a step:

TRL logs the reward, not each of a task's scores, so which turn improved isn't in the logs: an environment that owns its reward has no way to log metrics of its own.

What's supportedCopy Anchor Link

LimitsCopy Anchor Link

• TRL 1.15. Unsloth 2026.10.11 caps TRL at 1.15.0, so TRL's later features wait for Unsloth.

• One GPU tested. Every run here used one GPU; Unsloth's multi-GPU training wasn't tried.

• An answer in plain text reaches the task empty. The model answers through end_turn, as with TRL's own trainer.

• The reward, not each score. TRL calls an environment's get_reward() with no way to log metrics, so a task's separate scores aren't logged.

• A rollout isn't played again. One the env failed is left out of its step.

• Thinking takes room. max_completion_length covers the whole conversation. With thinking on, Qwen3-4B's reasoning on a GridSim day ran past 6,144 tokens before any tool call, so every rollout scored 0; on the inbox task it reasoned in about 500 tokens and called its tools.

• Local Docker runs out of networks past about 30 episodes. With the local sandbox, each episode of an env that runs in a container gets a Docker network of its own, and Docker's default address pools hold about 30. Add wider pools to Docker's daemon.json, and restart Docker before starting the trainer:

  Copy Text

  { "default-address-pools": [{ "base": "10.200.0.0/14", "size": 24 }] }