AsyncGRPOTrainer is stable: async RL in TRL

Introduction

AsyncGRPOTrainer has spent the last months in trl.experimental, where teams have used it for training coding agents that run in remote sandboxes, for the RL stage of a genomic model and more (see who uses it). With this release its API stays fixed. Here we focus on using it: running it, what a run looks like and what you can train with it. We’ll cover the internals and the design choices behind them in upcoming posts.

Why async RL

GRPO trains a model by sampling several answers to the same prompt, scoring them, and pushing the model toward the answers that scored better than their siblings. In TRL’s GRPOTrainer, that happens in two phases that take turns. First the model generates answers for a batch of prompts, then it trains on them. So the trainer GPUs sit idle while vLLM generates, and the other way around. And generation lasts as long as the slowest answer in the batch, which with reasoning models or agents can be many times longer than the average one. For example, in the RL run behind Reflection AI’s Beam, the median rollout took about 12 minutes, but in roughly one training step out of five at least one rollout took more than six hours.

Asynchronous RL puts generation and training on separate GPUs and lets them run at the same time. In TRL, that is AsyncGRPOTrainer. A rollout worker sends prompts to a vLLM server, scores the answers and puts them in a queue. The trainer takes whatever is in the queue, trains on it, and sends the new weights back to vLLM. Nobody waits for a whole batch to finish, but some answers are now generated by weights that are a few training steps old, and the trainer corrects for that. The animation below follows one rollout through that loop.

Labs that train this way report large gains. GLM-5 credits its “new asynchronous reinforcement learning infrastructure” with improving post-training efficiency “drastically”, and Olmo 3 reports that updating weights without pausing generation made training “up to 4x faster with the same resources, without hurting accuracy”.

When to use it

Use AsyncGRPOTrainer when generation is long or uneven, as with reasoning models and agents, because that is when a synchronous trainer waits the most. It needs vLLM on its own GPUs, so at least two. When answers are short and similar in length, there is less idle time to win back, and GRPOTrainer is usually enough. GRPOTrainer uses the same group-relative advantages and can also run vLLM on the same GPUs as training.

What a run looks like compares the two on the same task and GPUs.

For the wider landscape, Keep the Tokens Flowing compares 16 open-source libraries that train asynchronously.

Quick start

This is the quick start in the TRL docs. It trains Qwen3.5-2B on math problems, with a reward of 1 when the final answer is correct:

# train_async_grpo.py
from datasets import load_dataset
from trl import AsyncGRPOTrainer
from trl.rewards import accuracy_reward

if __name__ == "__main__":  # the rollout worker runs in its own process, which re-imports this file
    dataset = load_dataset("trl-lib/DeepMath-103K", split="train")

    trainer = AsyncGRPOTrainer(
        model="Qwen/Qwen3.5-2B",
        reward_funcs=accuracy_reward,
        train_dataset=dataset,
    )
    trainer.train()

The trainer needs vLLM and kernels, and accuracy_reward needs math-verify. Install them with:

pip install 'trl[vllm,kernels,math_verify]'

The vLLM server and the trainer run on separate GPUs. With two:

# Terminal 1: vLLM server on GPU 0
CUDA_VISIBLE_DEVICES=0 VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen3.5-2B \
    --max-model-len 4096 \
    --logprobs-mode processed_logprobs \
    --api-server-count 8 \
    --weight-transfer-config '{"backend":"nccl"}'

# Terminal 2: trainer on GPU 1
CUDA_VISIBLE_DEVICES=1 python train_async_grpo.py

VLLM_SERVER_DEV_MODE=1 turns on the endpoints the trainer uses to pause vLLM and load new weights, and --weight-transfer-config sends those weights over NCCL. --logprobs-mode processed_logprobs makes vLLM return each token’s log-probability after temperature, top-k and top-p are applied, which the trainer compares with its own. --api-server-count 8 keeps the extra log-probabilities requested by the default loss (score centering) cheap to serve, on vLLM 0.32 or later. --max-model-len should cover the longest prompt plus answer you expect. The same commands run on AMD GPUs, and vision-language models like Qwen3.5 train on text-only datasets (details).

To change the defaults, pass an AsyncGRPOConfig:

from trl import AsyncGRPOConfig

if __name__ == "__main__":
    ...
    args = AsyncGRPOConfig(
        output_dir="async-grpo-qwen3.5-2b",
        max_steps=200,
        max_completion_length=1024,
        report_to="trackio",  # needs pip install trackio
    )
    trainer = AsyncGRPOTrainer(model="Qwen/Qwen3.5-2B", args=args, reward_funcs=accuracy_reward, train_dataset=dataset)

These are only a few of its arguments. The AsyncGRPOConfig reference lists them all, including the ones inherited from TrainingArguments.

The trainer can also use more than one GPU, launched with accelerate launch. To shard the model across them, use FSDP2, since DeepSpeed isn’t supported (compatibility and constraints). For a complete script, see async_grpo_math.py.

What a run looks like

To see the difference, we trained Qwen3.5-2B on DeepMath twice, once with GRPOTrainer and once with AsyncGRPOTrainer, on the same two H200 GPUs (one for vLLM, one for training). Both used the same learning rate, 8 answers per prompt and up to 2,048 tokens per answer. GRPOTrainer trained on 64 answers per step and AsyncGRPOTrainer used gradient_accumulation_steps=16, and the benchmark script has the rest of the setup.

In 45 minutes, AsyncGRPOTrainer took 215 optimizer steps against 87, and vLLM generated about 3 times as many tokens (about 29 million against 9.3 million). Its reward stayed higher over the run, around 0.43 in the last 5 minutes against 0.25 for GRPOTrainer. This is one run of each, so the reward curves are noisy. The async run took its first step later, at 2.7 minutes against 0.6, because it fills the queue before training, and its samples were on average about 2 steps old, which the trainer corrects for.

The steps hold a similar number of useful answers. In about half of the groups, all 8 answers got the same reward, which gives no learning signal. GRPOTrainer still trains on them, so only about 33 of its 64 answers per step carry signal. AsyncGRPOTrainer drops those groups before training, so its steps hold about 25 answers, all with signal. That filter is part of the trainer, separate from running async. Many answers in both runs were cut off at 2,048 tokens, which keeps both rewards low. To see which side holds a run back, the docs explain the concurrency and rollout queue metrics.

What you can train with it

The first cases below build on the quick start and change a few lines of it.

Your own reward functions

A reward function receives the prompts and the completions as chat messages, their token IDs (completion_ids), plus every dataset column as a keyword argument, and returns one float per completion. With several functions, their rewards are added up. They can also be async, which helps when a reward calls a judge or an API.

def boxed_format(completions, **kwargs):
    """A small bonus when the final answer is written in \\boxed{}."""
    return [0.1 if "\\boxed{" in c[-1]["content"][-200:] else 0.0 for c in completions]

if __name__ == "__main__":
    ...
    trainer = AsyncGRPOTrainer(
        model="Qwen/Qwen3.5-2B",
        reward_funcs=[accuracy_reward, boxed_format],
        train_dataset=dataset,
    )

Reward functions run in the rollout worker, a separate process without GPUs, so define them at module level and keep them light. To score with a GPU model, serve it behind its own inference server and call it from the function (details).

Tools and environments

For multi-turn tool use, pass Python functions as tools, or an environment class as environment_factory. The trainer then builds one environment per rollout and exposes its other public methods to the model as tools. It calls reset() at the start, which can return the prompt, and get_reward() at the end if the environment defines it.

import random

from trl import AsyncGRPOConfig, AsyncGRPOTrainer

class IncrementEnv:
    def reset(self, **kwargs) -> str:
        self.counter, self.target = 0, random.randint(1, 6)
        return f"Increment the counter by {self.target}."

    def get_reward(self) -> float:
        return float(self.counter == self.target)

    def increment(self, step: int) -> int:
        """
        Increment the internal counter.

        Args:
            step: Value to add to the counter.

        Returns:
            The updated counter value.
        """
        self.counter += step
        return self.counter

if __name__ == "__main__":
    trainer = AsyncGRPOTrainer(
        model="Qwen/Qwen3-0.6B",
        args=AsyncGRPOConfig(max_steps=1000, chat_template_kwargs={"enable_thinking": False}),
        environment_factory=IncrementEnv,
    )
    trainer.train()

This example uses Qwen/Qwen3-0.6B, so restart the vLLM server with that model. Tools need type hints and a docstring, which is what the model reads. For real environments, the OpenEnv guide covers the same environment_factory contract with games like Wordle and Sudoku, and the agent training docs cover datasets and several environments in one run.

Agents in real harnesses

An off-the-shelf agent like OpenCode, Claude Code or pi runs its own tool loop in a sandbox, so the trainer can’t drive it. Instead, every model call the agent makes goes through a proxy to the same vLLM server, which records the exact token IDs and log-probabilities the trainer then trains on.

Start from the async_grpo_harbor example. Tasks come from Harbor, OpenEnv serves them and records the agent’s model calls, and a launcher starts the OpenEnv server, vLLM and the trainer on a single HF Job. The ultimate guide to multi-harness RL walks through it step by step.

The harness-hub project trained nine LoRA “harness adapters” this way, for three models (LFM2.5-24B-A2B, LFM3-1.7B and TwIL-8B) inside three agents (pi, opencode and hermes). The adapters’ model cards report up to 43% fewer tool calls than the base model on data-analysis tasks held out from training. The three LFM3-1.7B adapters also solve 10.7 to 16.0 points more tasks, while TwIL-8B and LFM2.5-24B-A2B stay about as accurate. Every checkpoint is on their live dashboard, and their write-up covers the setup.

LoRA

Add a peft_config to train an adapter instead of the full model (this needs pip install 'trl[peft]'):

from peft import LoraConfig

if __name__ == "__main__":
    ...
    trainer = AsyncGRPOTrainer(
        model="Qwen/Qwen3.5-2B",
        reward_funcs=accuracy_reward,
        train_dataset=dataset,
        peft_config=LoraConfig(r=32, lora_alpha=64, target_modules="all-linear"),
    )

By default, the trainer merges the adapter into the base model and sends vLLM the full weights (merged sync). To send only the adapter, start vLLM with LoRA enabled:

CUDA_VISIBLE_DEVICES=0 VLLM_SERVER_DEV_MODE=1 VLLM_ALLOW_RUNTIME_LORA_UPDATING=1 vllm serve Qwen/Qwen3.5-2B \
    --max-model-len 4096 \
    --logprobs-mode processed_logprobs \
    --weight-transfer-config '{"backend":"nccl"}' \
    --enable-lora \
    --max-lora-rank 32 \
    --max-loras 6

The adapter is about 1% of the bytes of the full weights. vLLM loads adapters at runtime only with a single API server, so this command has no --api-server-count 8, and the extra log-probabilities of score centering then cost generation throughput unless you set score_centering=False. vLLM also generates 1.33 to 1.39× slower per token with an adapter, so if generation is your bottleneck, merged sync is often still better. The docs cover --max-lora-rank, --max-loras and where adapters are written.

Going further

To change how rollouts are generated or scored, subclass AsyncRolloutLoop, override one of its methods (for example filter_prompt to skip prompts the model already solves) and set it as loop_cls on a subclass of the trainer. To train with a different objective, override the trainer’s compute_loss (guide). The trainer and vLLM also don’t have to share a machine. Async GRPO with LoRA across HF Jobs runs them on separate machines, and Delta Weight Sync sends only the weights that changed between steps through a Hub bucket.

Who uses it

At Hugging Face, it has been used in the RL stage of Carbon, the genomic foundation model, and for the harness-hub coding-agent adapters above. FineEnvs, an open project for building and training on RL environments for LLM agents, uses it to train models inside several agent harnesses at once. Outside, Axolotl runs its async GRPO on it, Unsloth plugs its own inference engine into it for LoRA training, and Snowflake’s Arctic Platform has RL recipes on top of it.

Wrapping up

AsyncGRPOTrainer is now stable. To try it, start from the quick start, or from the Harbor example if you want to train an agent.

In the coming days and weeks, we’ll publish more along these lines, including how the trainer controls staleness and concurrency, how it packs rollouts as a prefix tree, the loss and score centering, weight sync with and without LoRA, and benchmarks against other libraries.

We’d love to see what you train with it. Share your runs and models in the comments, open an issue if something doesn’t work, and tell us which of these topics you’d like us to cover first.

Happy training!