# Unsloth (https://www.agentenvframework.com/docs/training/unsloth)

> Train using Unsloth

## How a rollout runs

1. Unsloth’s `GRPOTrainer` takes a row of `env.dataset`, one task of the split, and resets an environment from `env.factory` with it, as TRL’s does.
2. The server starts the row’s task as an episode: it deploys the env, loads the task, and opens the agent’s turn, while Unsloth’s vLLM, sharing the model’s weights, writes its first reply.
3. Your model plays the turn: each tool call TRL parses is a method on the environment, which runs it on the env through the episode.
4. Your model ends its turn with `end_turn`, and the task’s verifier scores the episode, as it would score any agent’s run.
5. `get_reward()` hands the score to the trainer as the reward; the episode is torn down, and the session takes the next row.

Parts of the scene:

- **Unsloth**: Unsloth’s `GRPOTrainer`, TRL’s rewritten when `unsloth` is imported first, with `train_dataset=env.dataset` and `environment_factory=env.factory`. With `fast_inference=True`, it generates on a vLLM in its own process that shares the model’s weights. It owns the model, the generation and the update; the environment owns the reward.
- **A dataset row**: One task of the split: its prompt as chat messages, and the `split`, `index` and `task_id` that `reset` plays. The prompt is yours to change; the task stays the same.
- **An environment**: An instance of the class `env.factory` builds, holding one session on the server. Its methods are the turn’s tools plus `end_turn`; `reset` starts a task and `get_reward` reads its score. TRL reuses it rollout after rollout.
- **The session**: One OpenEnv WebSocket session per environment, so one episode at a time, held from one rollout to the next. A session a restarting server dropped opens again once it’s back, within `reset_timeout`.
- **The OpenEnv server**: `agent_env.openenv.build_app(bundle)` serves every task of a bundle as an OpenEnv env, the split `all` plus one per eval. `AgentEnv.local(bundle)` starts it in the trainer’s own process.
- **The env**: The task’s env, deployed fresh for this episode in a sandbox, here a container on your Docker. Its tools are what the model calls.
- **The agent’s turn**: The task prompts one agent, and your model plays it from outside: the episode deploys a proxy in the agent’s place and hands the turn’s prompt and tools to the environment.
- **The verifier**: The task’s own verifier step scores the episode once the turn ends, the same score an evaluation run records. With several scores, `reward=` picks the one to train on.
- **The reward**: `get_reward()` returns the score, or `None` when the episode ended in an error, which Unsloth’s trainer, like TRL’s, leaves out of the update and the group’s baseline.

[Unsloth](https://github.com/unslothai/unsloth)'s `GRPOTrainer` is TRL's, rewritten when `unsloth` is
imported first. With `fast_inference=True` it generates on a vLLM in its own process that shares the
model's weights, so GRPO on a 4B model fits one GPU, in 4-bit a 24 GB one. It takes the
[TRL helper](https://www.agentenvframework.com/docs/training/trl.md)'s dataset and environments as they are:

| Piece                                 | What it does                                                                                                                                                                                          |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `agent_env.openenv.build_app(bundle)` | Serves every task of a bundle as an [OpenEnv](https://www.agentenvframework.com/docs/openenv.md) env: the split `all`, plus one split per eval. `AgentEnv.local(bundle)` starts it in the trainer's process.                              |
| `agent_env.training.trl.AgentEnv`     | Reads the server's tasks into `env.dataset`, and makes `env.factory`, the `environment_factory` whose environments play them: their methods are the tools, and `get_reward()` is the episode's score. |

Unsloth's tokenization, generation and training stay its own, and it calls no reward function of
yours: the task's verifier scores each episode.

## Install

You need Linux with an NVIDIA GPU, Python 3.11 or newer, and Docker running:

```bash title="Terminal"
pip install 'agentenv-framework[trl]>=0.9.1407' 'unsloth==2026.10.11' 'unsloth_zoo==2026.10.11' \
  'trl==1.15.0' 'transformers==5.17.0' 'vllm==0.29.0' 'vllm-bnb-plugin==0.0.3'
```

> [!WARNING]
> **Unsloth's pins**
>
> * **TRL and transformers.** Unsloth 2026.10.11 caps TRL at 1.15.0 and transformers at 5.17.0. The
>   TRL helper needs TRL 1.15 or later, for an environment that owns its reward, so TRL 1.15.0 is the
>   one both take.
> * **vLLM.** vLLM 0.29 is the newest the helper takes (OpenEnv needs mcp 1, vLLM 0.30 mcp 2).
>   From vLLM 0.28, 4-bit models need `vllm-bnb-plugin` to load in vLLM.
> * **Import order.** Import `unsloth` before `trl`, so Unsloth rewrites TRL's trainers before your
>   script uses them.

## Train on GridSim

GridSim is the bundle the [TRL page](https://www.agentenvframework.com/docs/training/trl.md#train-on-gridsim) trains on. Each task runs
one grid battery for one day in the ERCOT day-ahead market: the model reads the day's 24 hourly
prices and submits a schedule, one integer an hour, to charge, sell or idle. The env grades the
schedule against the day's exact optimum: 0 for nothing useful, 0.2 for idling all day, 1 for the
optimum. The bundle has 2,000 train days and 200 eval days.

```bash title="Terminal"
hf download earakely-scale/GridSimEnv-AgentEnv --repo-type dataset --include "bundles/gridsim-v1/*" --local-dir .
```

### Warm start with SFT

Played as it comes, Qwen3-4B idles: it scores 0.18 on the 200 eval days. Warm-start it with the
TRL page's [`sft.py`](https://www.agentenvframework.com/docs/training/trl.md#warm-start-with-sft), in an environment set up as that page's
[Install](https://www.agentenvframework.com/docs/training/trl.md#install) has it, so it saves `qwen3-4b-gridsim-sft`. It took 25 minutes on
one L40S.

### Train with GRPO

This run uses the other trainers' runs' settings: the same SFT warm start, LoRA r32/α64 on every
linear layer, learning rate 2e-5 cosine with 3 warmup steps, 32 rollouts a step (4 days × 8) at
temperature 1.1, thinking off, and at most two tool-calling rounds a rollout. It sets back the
defaults Unsloth changes (under [Settings Unsloth changes](#settings-unsloth-changes)), so its row
compares with the TRL run's:

```python title="train.py"
from unsloth import FastLanguageModel  # first, so Unsloth rewrites TRL's trainers

from trl import GRPOConfig, GRPOTrainer

from agent_env.training.trl import AgentEnv

env = AgentEnv.local("bundles/gridsim-v1", split="train", max_concurrent_envs=32)
model, tokenizer = FastLanguageModel.from_pretrained(
    "qwen3-4b-gridsim-sft", max_seq_length=4096, load_in_4bit=False, fast_inference=True,
    max_lora_rank=32, gpu_memory_utilization=0.5)
model = FastLanguageModel.get_peft_model(
    model, r=32, lora_alpha=64,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    use_gradient_checkpointing="unsloth", random_state=0)
trainer = GRPOTrainer(
    model=model,
    processing_class=tokenizer,
    args=GRPOConfig(
        output_dir="out",
        max_steps=40,
        per_device_train_batch_size=8,
        gradient_accumulation_steps=4,  # 32 rollouts a step: 4 days, 8 rollouts each
        num_generations=8,
        max_completion_length=1024,
        max_tool_calling_iterations=2,
        chat_template_kwargs={"enable_thinking": False},
        temperature=1.1,
        learning_rate=2e-5,
        lr_scheduler_type="cosine",
        warmup_steps=3,
        loss_type="dr_grpo",
        scale_rewards=None,  # Unsloth turns reward scaling off for dr_grpo unless it's None
        beta=0.0,
        epsilon_high=0.28,
        optim="adamw_torch",  # AdamW, as TRL's adamw_torch_fused; Unsloth's is adamw_8bit
        weight_decay=0.0,  # TRL's default; Unsloth's is 0.001
        use_vllm=True,
        bf16=True,
        logging_steps=1,
        eval_strategy="steps",
        eval_steps=20,
        eval_on_start=True,
        num_generations_eval=1,
        per_device_eval_batch_size=32,
        save_strategy="no",
        seed=0,
        report_to="none",
    ),
    train_dataset=env.dataset,
    eval_dataset=env.dataset_for("eval"),
    environment_factory=env.factory,
)
trainer.train()
```

There are no `reward_funcs`: each environment owns its reward through `get_reward()`, which the
trainer reads once the rollout ends. TRL's `vllm_*` settings don't apply: Unsloth's vLLM takes
its memory and context from `from_pretrained`.

```json title="Output (step 12 of the GridSim run, trimmed)"
{"loss": 0.0523, "grad_norm": 0.0498, "reward": 0.6434, "reward_std": 0.3533, "rewards/gridsim-v1/mean": 0.6434,
 "tools/call_frequency": 2.0, "tools/failure_frequency": 0.0, "completions/mean_length": 681.2, "step_time": 101.8}
```

The reward column is named after the environment class, here the bundle's name. The server needs a
session for each rollout in flight: 32 a step, and 32 an eval batch.

### What it learns

| Qwen3-4B | eval reward (200 days) |
|---|---|
| after SFT (GRPO step 0) | 0.551 |
| GRPO step 20 | 0.520 |
| GRPO step 40 | 0.595 |

Mean train reward of each GRPO step's 32 rollouts, by step: 1: 0.628, 2: 0.848, 3: 0.660, 4: 0.757, 5: 0.532, 6: 0.629, 7: 0.499, 8: 0.491, 9: 0.826, 10: 0.747, 11: 0.480, 12: 0.643, 13: 0.514, 14: 0.653, 15: 0.748, 16: 0.611, 17: 0.771, 18: 0.596, 19: 0.907, 20: 0.451, 21: 0.619, 22: 0.598, 23: 0.727, 24: 0.688, 25: 0.689, 26: 0.695, 27: 0.635, 28: 0.736, 29: 0.745, 30: 0.715, 31: 0.712, 32: 0.500, 33: 0.618, 34: 0.610, 35: 0.687, 36: 0.621, 37: 0.537, 38: 0.804, 39: 0.840, 40: 0.420.

| Qwen3-4B, eval reward on the 200 eval days | Unsloth | TRL  | SkyRL | verl | prime-rl | NeMo RL |
| ------------------------------------------ | ------- | ---- | ----- | ---- | -------- | ------- |
| After SFT (GRPO step 0)                    | 0.55    | 0.57 | 0.54  | 0.52 | 0.56     | 0.55    |
| After 20 GRPO steps                        | 0.52    | 0.56 | 0.55  | 0.59 | 0.58     | 0.68    |
| After 40 GRPO steps                        | 0.59    | 0.61 | 0.61  | 0.60 | 0.57     | 0.67    |

All six start from the same SFT model and score it alike, within what one rollout a day can tell
apart (about ±0.06). Through Unsloth, as through TRL, the train reward stays level, at 0.66 over the
first 10 steps and over the last 20, and the eval ends at 0.59. GridSim's greedy rule scores 0.76 on
these days. This is one run of each trainer, so read the gaps as pointers, not a ranking.

The 40 GRPO steps with the three evals took 103 minutes on one L40S with 16 vCPUs, an AWS
g6e.4xlarge: about 104 seconds a step, and 9 to 11 minutes an eval. No rollout failed. The TRL run's
steps took about 190 seconds on a g6e.xlarge with 4 vCPUs, most of it deploying the envs, so the two
times compare machines as much as trainers.

## Settings Unsloth changes

Unsloth's `GRPOConfig` gives these arguments other defaults when you leave them unset, and changes
one you set:

| Argument                                   | Unsloth's                                     | TRL's               | Here                                  |
| ------------------------------------------ | --------------------------------------------- | ------------------- | ------------------------------------- |
| `optim`                                    | `adamw_8bit`                                  | `adamw_torch_fused` | `adamw_torch`, the same AdamW unfused |
| `weight_decay`                             | 0.001                                         | 0.0                 | 0.0                                   |
| `learning_rate`                            | 5e-5                                          | 1e-6                | 2e-5                                  |
| `warmup_steps`                             | 0.1 (10% of the steps)                        | 0                   | 3                                     |
| `seed`                                     | 3407                                          | 42                  | 0                                     |
| `per_device_eval_batch_size`               | 4                                             | 8                   | 32                                    |
| `scale_rewards` with `loss_type="dr_grpo"` | `False`, even when set to `"group"` or `True` | `"group"`           | `None`, which keeps TRL's scaling     |

The warmup counts from 0, so the first step trains at learning rate 0 unless `warmup_steps=0`.

## On a 24 GB GPU

The same SFT model, loaded in 4-bit, trained with the run's settings on one L4 with 24 GB, an AWS
g6.4xlarge, using about 22 GB of it. Only the loading changes:

```python title="train.py"
model, tokenizer = FastLanguageModel.from_pretrained(
    "qwen3-4b-gridsim-sft", max_seq_length=4096, load_in_4bit=True, fast_inference=True,
    max_lora_rank=32, gpu_memory_utilization=0.5)
```

| Qwen3-4B, GridSim                   | 16-bit on the L40S | 4-bit on the L4 |
| ----------------------------------- | ------------------ | --------------- |
| Eval reward after SFT (GRPO step 0) | 0.55               | 0.51            |
| After 10 GRPO steps                 | not evaluated      | 0.45            |
| A GRPO step of 32 rollouts          | 104 s              | 269 s           |
| An eval of the 200 days             | 9 to 11 min        | 30 to 31 min    |

The 4-bit model starts a little below the 16-bit one, and its 10 steps leave it within what one
rollout a day can tell apart. A 40-step run with three evals would take about four and a half hours
on the L4.

## Serve the bundle on its own host

To keep the envs off the GPU machine, serve the bundle where they can deploy, and point the trainer
at it. The rest of `train.py` stays the same:

```python title="serve.py, on the env host"
import uvicorn

from agent_env.openenv import build_app

app = build_app("bundles/gridsim-v1", max_concurrent_envs=48)
uvicorn.run(app, host="0.0.0.0", port=8000)
```

```python title="train.py, on the GPU machine"
env = AgentEnv("http://env-host:8000", split="train")
```

Each environment holds one session on the server, so `max_concurrent_envs` must cover every
rollout in flight, here the 32 of a step and the 32 of an eval batch. `build_app(..., sandbox="modal")`
deploys each episode's envs on another [sandbox provider](https://www.agentenvframework.com/docs/sandboxes.md) instead of the host's
Docker, for a GPU machine without Docker, such as a notebook's.

## Failed rollouts

`get_reward()` returns `None` when an episode ended in an error: a deploy that failed, a call past
its deadline, or a server that went away. Unsloth's trainer, like TRL's, leaves that rollout out of
the update and of its group's baseline, and logs it:

```text title="Log"
WARNING agent_env.training.trl: rollout left unscored: end_turn: ConnectionError: Failed to connect to ws://127.0.0.1:8200/ws: [Errno 111] Connect call failed ('127.0.0.1', 8200)
```

A rollout isn't played again: the step trains on the rollouts that scored. An environment holds its
session from one rollout to the next, and when the server restarts, the next rollout waits for it to
come back, within `reset_timeout`, rather than failing. In two short runs on the L4, with 16 rollouts
a step, the env server was stopped 10 seconds into step 2 and was back 30 seconds later:

|                                                | agentenv-framework 0.9.1406 and earlier | 0.9.1407                  |
| ---------------------------------------------- | --------------------------------------- | ------------------------- |
| Step 2's rollouts, mid-episode when it stopped | unscored                                | unscored                  |
| Step 3's, started while it came back           | unscored                                | waited for it, and scored |
| The step after                                 | not run                                 | scored                    |

A step whose rollouts all ended unscored trains on nothing, and TRL warns that every reward function
returned `None`.

## Multi-turn tasks

A task with several turns plays as one rollout: `end_turn` hands the model each next turn's prompt,
as on the [TRL page](https://www.agentenvframework.com/docs/training/trl.md#what-the-model-sees). The model's answer reaches the task
only through `end_turn(response=...)`, so a turn it answers in plain text ends with an empty reply,
which a verifier judging the reply scores as such: tell the model in the system prompt to answer with
`end_turn`. A task that records several scores needs a reward that combines them, here their mean:

```python title="train.py"
env = AgentEnv.local("bundles/my-bundle", reward=lambda scores: sum(scores.values()) / len(scores))
```

A test bundle of four two-turn questions about an inbox, with a regex verifier a turn, trained on the
L4 in 4-bit, Qwen3-4B, 5 GRPO steps of 32 rollouts at about 4 minutes a step:

|                                              | Step 1 | Step 5 |
| -------------------------------------------- | ------ | ------ |
| Train reward                                 | 0.16   | 0.25   |
| Tool calls a rollout                         | 3.4    | 4.6    |
| Eval reward, the four tasks before and after | 0.125  | 0.25   |

TRL logs the reward, not each of a task's scores, so which turn improved isn't in the logs: an
environment that owns its reward has no way to log metrics of its own.

## What's supported

|                               |                                                                                                                                                                           |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Unsloth                       | 2026.10.11, on TRL 1.15.0, transformers 5.17.0 and vLLM 0.29.0. The page's runs used it on one L40S and one L4.                                                           |
| Precision                     | 16-bit LoRA and 4-bit QLoRA (`load_in_4bit=True`, with `vllm-bnb-plugin`).                                                                                                |
| Generation                    | Unsloth's colocated vLLM (`fast_inference=True`), or transformers' `generate` without it.                                                                                 |
| Trainers                      | Unsloth's `GRPOTrainer`, with evals during training, and its `AsyncGRPOTrainer`, which on one GPU generates on the model's own vLLM rather than a server's.               |
| Models                        | Qwen3 and Qwen3.5: TRL parses their tool calls, with thinking on or off.                                                                                                  |
| Multi-turn tasks              | `end_turn` hands the model each next prompt.                                                                                                                              |
| Bundles with several toolsets | `env.factory` is a dict of classes, one per toolset, and TRL routes each row to its own: rows of GridSim and of the inbox bundle, mixed in one dataset, trained together. |
| Envs                          | `AgentEnv.local(bundle)` in the trainer's process, or `AgentEnv(url)` against a server on another host, whose envs deploy on any [sandbox provider](https://www.agentenvframework.com/docs/sandboxes.md).     |

## Limits

* **TRL 1.15.** Unsloth 2026.10.11 caps TRL at 1.15.0, so TRL's later features wait for Unsloth.
* **One GPU tested.** Every run here used one GPU; Unsloth's multi-GPU training wasn't tried.
* **An answer in plain text reaches the task empty.** The model answers through `end_turn`, as with
  TRL's own trainer.
* **The reward, not each score.** TRL calls an environment's `get_reward()` with no way to log
  metrics, so a task's separate scores aren't logged.
* **A rollout isn't played again.** One the env failed is left out of its step.
* **Thinking takes room.** `max_completion_length` covers the whole conversation. With thinking on,
  Qwen3-4B's reasoning on a GridSim day ran past 6,144 tokens before any tool call, so every rollout
  scored 0; on the inbox task it reasoned in about 500 tokens and called its tools.
* **Local Docker runs out of networks past about 30 episodes.** With the `local` sandbox, each
  episode of an env that runs in a container gets a Docker network of its own, and Docker's default
  address pools hold about 30. Add wider pools to Docker's `daemon.json`, and restart Docker before
  starting the trainer:

  ```json title="/etc/docker/daemon.json"
  { "default-address-pools": [{ "base": "10.200.0.0/14", "size": 24 }] }
  ```