Part 1 of a series on practical post-training pipelines for deployed agents.

Deployed agents spend compute in two places. Training runs take hours, sometimes days. Inference takes seconds per turn, multiplied across users and retries. You have to decide where to pay.

This series looks at that tradeoff on real benchmarks, with code, datasets, and checkpoints for each pipeline.

This post uses Tau2-bench, a multi-turn tool-use benchmark where an agent guides a user through 20+ turns of diagnostics to resolve a specific issue. The task reward arrives only at the final step, so credit assignment is the hard part.

I trained Qwen3-4B with SFT, then rejection fine-tuning (RFT) on teacher trajectories, then GRPO. On the 100-task test split, GRPO reached 59% Pass@4 and SFT1, the checkpoint before GRPO, reached 60% with the same sampling setup. I read that as a close comparison, not a ranking. Below is the recipe, and what this result changed in how I value another training stage.

Evaluation scope. Pass@k means at least one success among k full attempts, not the pass^k leaderboard estimate. Pass@1 is greedy unless labeled sampled. The benchmark oracle scores each attempt. Training took about 5.5 hours on 4×H100 (about 22 GPU-hours), and Pass@4 evaluation about two hours on 2×H100.

Tau2-bench performance chart comparing larger baseline models to a 4B model, highlighting the Pass@1 vs Pass@4 tradeoff under verifier-based sampling

Tau2-bench performance, all domains. The highlighted slot shows GRPO at Pass@1 and SFT1 at Pass@4 with four sampled attempts.

Training data · SFT1 checkpoint · GRPO checkpoint

Where the Work Is

Most of the effort in these pipelines goes into the SFT data. If the model cannot follow the protocol, it cannot generate useful rollouts, and RFT and GRPO have little to optimize. My baseline is a model that follows the protocol, plus a verifier I trust to score its rollouts.

For Tau2-bench, the SFT data has to teach basics that are easy to skip:

  • Valid tool calls and arguments across 30+ tools
  • Turn discipline (one action, then wait for the observation)
  • Dual-control language (coach the user, do not pretend you can click buttons)
  • Recovery behavior when something fails or is ambiguous

Context: Credit Assignment in Multi-Turn Tool-Use

In a telecom troubleshooting run, the agent might ask the user to grant app permissions at step 15. That action decides the outcome, but the reward arrives only at the final step. A binary outcome reward updates every sampled action, yet over a 20-turn interaction it says little about which turn caused success or failure.

How Do We Do Multi-Turn RL?

Stage 1: SFT (Teaching Protocol)

Multi-turn tool-use agent training pipeline showing SFT, expert iteration (RFT to SFT1), optional GRPO for higher Pass@1, and a deployment path using verifier-based selection

Training and deployment pipeline. The matched comparison tests whether GRPO adds value beyond four sampled attempts. Deploying with selection requires a verifier.

In my setup, SFT taught the protocol: valid tool calls, turn-taking, and dual control, where the agent coaches users through diagnostics instead of running them. On its own, SFT scored below the baseline (8.6% vs 14.3%), but it produced a model that could generate usable rollouts for filtering and RL. The gains came from the next stage.

Stage 2: Rejection Fine-Tuning (RFT)

After SFT, the model completes tasks inconsistently. Sampling several rollouts and keeping the successes concentrates training on strategies that work. The published tau2-sft-seed-v3 dataset holds the 839 kept trajectories. They come from teacher models rather than the SFT checkpoint: Qwen3-235B-A22B in every domain, plus Claude Opus 4.5 for 150 telecom trajectories, against a gemini-2.5-flash-lite user simulator. The set keeps successes plus 266 partial-credit trajectories with a partial score of at least 0.52.

RAFT shows that this kind of rejection sampling on verifier rewards can approach GRPO with faster early learning, and that much of the gap comes from sample selection, such as filtering prompts where every sampled response is wrong. It also shows the limit of training only on successes: entropy collapses quickly, exploration dries up, and the gap between Pass@1 and Pass@k grows. RL can still help when Pass@1 matters or sampling is expensive.

In my runs, GRPO’s gains were much larger at Pass@4 than at greedy Pass@1. That raises a practical question: if you have a verifier at deployment, is RL worth the training cost?

Stage 3: GRPO + Turn-Level Reward Shaping

GRPOGRPO (Group Relative Policy Optimization): samples K trajectories per prompt, scores them, and raises the probability of actions that scored above the group average. It needs no separate critic network. samples K trajectories per prompt and updates each sampled action by its reward relative to the group. With an outcome reward, every action in a trajectory gets the same advantage, which does not identify the turn that mattered.

To add that signal, I shaped the reward with Tau2-bench’s turn-level checks (action checks, communication checks, environment assertions):

shaped_reward = task_reward + alpha * partial_score

The partial score adds feedback on intermediate progress beyond the final binary outcome.

Results

Stage Tasks Overall Airline Retail Telecom
Baseline (Qwen3-4B-Instruct) 70 14.3% 5.0% 16.0% 20.0%
SFT 70 8.6% 5.0% 20.0% 0.0%
SFT1 100 27.0% 20.0% 50.0% 7.5%
GRPO (Pass@1, greedy) 70 32.9% 15.0% 76.0% 4.0%
GRPO (Pass@4) 70 57.1% 50.0% 76.0% 44.0%
SFT1 (Pass@4) 100 60.0% 30.0% 82.5% 52.5%

Training logs (WandB)

Rows are greedy Pass@1 unless labeled Pass@4. Pass@4 counts a task as solved if any of four sampled attempts passes the benchmark oracle.

Correction, October 17. This table mixes two task sets. The per-domain percentages in the rows marked 100 fit only the pinned 100-task test split (20 airline, 40 retail, 40 telecom). The rows marked 70 do not fit the full split; their percentages fit a 70-task subset of 20 airline, 25 retail, and 25 telecom tasks. Compare rows only within the same task set. The matched comparison below uses the 100-task split for both models.

Comparing SFT1 and GRPO under the same sampling setup

I evaluated SFT1 with the same sampling setup as GRPO’s Pass@4. Here Pass@1 is the first sampled attempt and Pass@4 is at least one success among four, so these Pass@1 numbers differ from the greedy ones above.

Model Pass@1 Pass@4
SFT1 29% 60%
GRPO 36% 59%

GRPO improved sampled Pass@1. At Pass@4, the two were within one point. Sampling is a useful baseline before adding RL, if the application can afford four full attempts and can identify a successful one. Interactive tasks may also need isolated retries so a new attempt does not repeat real actions.

Eval config

Eval command hyperparameters:

  • --domains airline,retail,telecom
  • --task-split test
  • --num-samples 4
  • --temperature 0.8
  • --top-p 1.0
  • --top-k 20

Environment variables:

  • TAU2_USE_COMPRESSED_PROMPTS=0
  • TAU2_USER_MODEL=gpt-4.1-mini
  • TAU2_USER_TEMPERATURE=0.7

Policy server (sglang):

  • --model-path Jarrodbarnes/Qwen3-4B-tau2-sft1 (for SFT1 eval) or Jarrodbarnes/Qwen3-4B-tau2-grpo-v1 (for GRPO eval)
  • --tp 1
  • --mem-fraction-static 0.70
  • --port 30000

What Didn’t Work

Pure SFT made things worse. Training on unfiltered trajectories dropped accuracy from 14.3% to 8.6%. Following the protocol did not by itself produce task success.

Telecom is still the hardest domain. At GRPO Pass@4 on the 70-task set, retail reaches 76% and telecom 44%. On the 100-task split, SFT1 Pass@4 reaches 82.5% on retail and 52.5% on telecom. When the agent has to talk the user through physical actions instead of executing them, errors compound across turns.

Implementation Notes

Dual-control (telecom): Diagnostic actions are user-only, so the agent instructs instead of executing:

Agent: "Please toggle airplane mode ON, wait 10 seconds, then OFF."
User: "Done. Still no data."

Function calling: Qwen3 uses <tool_call>{...}</tool_call>. Include </tool_call> in stop sequences.

User simulator: Training uses a local instruct model on port 30001. Evaluation uses GPT-4.1-mini.

Resources

Code SFT1 Checkpoint GRPO Checkpoint Dataset Benchmarks

The question I started with, where compute should go, has a messier answer than I expected. GRPO improved single-attempt success, while four sampled attempts brought SFT1 level with it at Pass@4. The deployment decision depends on what those attempts cost and whether a real verifier can recover the successes Pass@4 counts.