Part 1 of a series on practical post-training pipelines for deployed agents.

When you deploy agents in enterprise environments, compute spend shows up as time. Training runs take hours, sometimes days. Inference takes seconds per turn, multiplied across users and retries. You have to decide where you want to pay.

This series is a set of practical posts on how I think about that tradeoff. Each post uses a real benchmark or use case and a concrete pipeline, with code, datasets, and checkpoints.

I’ll start with Tau2-bench, a multi-turn tool-use benchmark where an agent guides a user through 20+ turns of diagnostics before solving a specific issue. Credit assignment is what breaks. The task reward only arrives at the final step.

I trained Qwen3-4B using SFT → rejection fine-tuning (RFT) → GRPO. One representative run reached 57.1% Pass@4, and the best GRPO run reached 59%. I walk through the recipe below, plus what changed my mind about the value of another training stage.

I ran a Pass@4 ablation on SFT1, the checkpoint before GRPO. With the same sampling setup, it reached 60%. I would call that comparison close rather than rank the checkpoints on a one-point gap.

Evaluation scope. Pass@k here means any success among k full attempts, not the pass^k leaderboard estimate. Unless labeled sampled, Pass@1 is greedy. Success is scored by the benchmark oracle after each attempt. Rough compute budget was two hours on 8×H100 for training and two hours on 2×H100 for Pass@4 evaluation.

Tau2-bench performance chart comparing larger baseline models to a 4B model, highlighting the Pass@1 vs Pass@4 tradeoff under verifier-based sampling

Tau2-bench performance (all domains). The highlighted slot shows Pass@1 for GRPO and Pass@4 for SFT1 plus verifier-based sampling (k=4).

Training data · SFT1 checkpoint · GRPO checkpoint

Where the Work Is

In these pipelines, most of the effort goes into the SFT data. If the model cannot follow the protocol, it cannot generate useful rollouts, and neither RFT nor GRPO has much to optimize. I treat the baseline as a model that can follow the protocol plus a verifier that can score rollouts in a way you trust.

For tau2-bench, good SFT data needs to teach basics that are easy to gloss over:

  • Valid tool calls and arguments across 30+ tools
  • Turn discipline (one action, then wait for the observation)
  • Dual-control language (coach the user, do not pretend you can click buttons)
  • Recovery behavior when something fails or is ambiguous

RFT is a practical middle step. You sample, score with a verifier, and train on the trajectories that worked. The RAFT paper shows how far this can go with verifiable rewards, and how much of the perceived gap to GRPO comes down to sample selection (for example, avoiding prompts where every sampled rollout is wrong).

The same paper also shows a limitation of positive-only selection. Entropy collapses fast. Early gains come quickly, then exploration dries up. In practice, that shows up as a bigger difference between Pass@1 and Pass@k. RL can still help when you care about Pass@1, or when sampling is expensive.

Context: Credit Assignment in Multi-Turn Tool-Use

In a telecom troubleshooting run, the agent might ask the user to grant app permissions at step 15, a critical action. But the task reward only arrives at the final step.

How does the model learn that step 15 mattered?

A binary outcome reward can update every sampled action, but it gives little information about which turn caused success or failure. Over a 20-turn interaction, that makes credit assignment difficult.

How Do We Do Multi-Turn RL?

Stage 1: SFT (Teaching Protocol)

Multi-turn tool-use agent training pipeline showing SFT, expert iteration (RFT to SFT1), optional GRPO for higher Pass@1, and a deployment path using verifier-based selection

Training and deployment pipeline for multi-turn tool-use agents. The matched comparison tests whether the GRPO stage adds value beyond four sampled attempts. A deployment path using selection also requires an available verifier.

Before a model can optimize tool-use, it needs the basics: one action per turn, then wait. Thirty-plus tools with complex argument structures. And in telecom, a constraint that trips up most systems - the agent coaches users through diagnostics rather than executing them directly.

Without SFT, RL training will struggle to find any learning signal. In our setup, SFT mainly teaches the protocol (valid tool calls, turn-taking, and dual-control). On its own, it underperformed the baseline (8.6% vs 14.3%), but it produced a model that could generate usable rollouts for filtering and RL.

In our runs, pure SFT was not enough. The real gains came once we started filtering for successful trajectories (Stage 2).

Stage 2: Rejection Fine-Tuning (RFT)

After SFT, the model can complete tasks but does so inconsistently. Sampling multiple rollouts and keeping only successes concentrates the training distribution on viable strategies. The filtering is simple: sample 4-8 attempts at temperature 0.8, keep trajectories that succeed (reward >= 1.0), and for tasks with no successes, keep the highest partial score if it clears 0.6.

Recent work on RAFT shows that RFT-style training (rejection sampling on verifier rewards) can approach GRPO performance with faster early-stage learning, and that a large part of the gap comes down to sample selection and exploration dynamics (e.g., filtering prompts where all sampled responses are wrong). Combined with our observation that GRPO’s gains are much larger at Pass@4 than at greedy Pass@1, this raises a practical question: if you have a verifier at deployment, is RL worth the training cost?

The published tau2-sft-seed-v3 dataset results from this filtering.

Stage 3: GRPO + Turn-Level Reward Shaping

GRPOGRPO (Group Relative Policy Optimization): samples K trajectories per prompt, scores them, and trains the model to increase probability of high-reward actions relative to the group average. No separate critic network required. provides a group-relative learning signal. I combined it with turn-level reward shaping:

Group-based advantage estimation. For each prompt, sample K trajectories, score them, and use their rewards relative to the group to update the sampled actions. With an outcome reward, all actions in a trajectory share the same advantage; that alone does not identify which turn mattered.

Dense reward shaping: Tau2-bench provides turn-level evaluation (action checks, communication checks, environment assertions). We extract partial scores and shape rewards:

shaped_reward = task_reward + alpha * partial_score

The partial score adds feedback about intermediate progress beyond the final binary outcome.

Results

Stage Overall Airline Retail Telecom
Baseline (Qwen3-4B-Instruct) 14.3% 5.0% 16.0% 20.0%
SFT 8.6% 5.0% 20.0% 0.0%
SFT1 27.0% 20.0% 50.0% 7.5%
GRPO (Pass@1, greedy) 32.9% 15.0% 76.0% 4.0%
GRPO (Pass@4) 57.1% 50.0% 76.0% 44.0%
SFT1 + verifier (Pass@4) 60.0% 30.0% 82.5% 52.5%

Training logs (WandB)

All rows are greedy (Pass@1) unless otherwise noted. Pass@4 uses sampling with a verifier.

Comparing SFT1 and GRPO under the same sampling setup

I ran the ablation I left open earlier. I evaluated the SFT1 checkpoint with the same sampling setup used for Pass@4. Pass@1 here means the first sampled attempt. Pass@4 means at least one success among 4 attempts. These Pass@1 numbers are sampled, so they will not match the greedy Pass@1 results above.

SFT1 vs GRPO (same sampling setup)

Model Pass@1 Pass@4
SFT1 29% 60%
GRPO 36% 59%

GRPO improved sampled Pass@1 in this comparison. At Pass@4, the checkpoints were close. This makes sampling a useful baseline before adding RL, provided the application can afford four full attempts and identify a successful one. Interactive tasks may also require isolated retries to avoid repeating real actions.

Eval config

Eval command hyperparameters:

  • --domains airline,retail,telecom
  • --task-split test
  • --num-samples 4
  • --temperature 0.8
  • --top-p 1.0
  • --top-k 20

Environment variables:

  • TAU2_USE_COMPRESSED_PROMPTS=0
  • TAU2_USER_MODEL=gpt-4.1-mini
  • TAU2_USER_TEMPERATURE=0.7

Policy server (sglang):

  • --model-path Jarrodbarnes/Qwen3-4B-tau2-sft1 (for SFT1 eval) or Jarrodbarnes/Qwen3-4B-tau2-grpo-v1 (for GRPO eval)
  • --tp 1
  • --mem-fraction-static 0.70
  • --port 30000

What Didn’t Work

Pure SFT made things worse in this run. Training on unfiltered trajectories dropped accuracy from 14.3% to 8.6%. Protocol fluency alone did not translate into task success.

Telecom is still the hardest domain. Retail reaches 76% while telecom stays at 44%. When the agent must instruct users through physical actions rather than execute them directly, error propagation compounds across turns.

Sparse rewards made credit assignment difficult. With 20+ turn episodes, binary outcomes did not identify which intermediate decisions helped. I used turn-level partial scores to provide more specific feedback.

Implementation Notes

Dual-control (telecom): Diagnostic actions are user-only. The agent instructs rather than executes:

Agent: "Please toggle airplane mode ON, wait 10 seconds, then OFF."
User: "Done. Still no data."

Function calling: Qwen3 uses <tool_call>{...}</tool_call>. Include </tool_call> in stop sequences.

User simulator: Training uses a local instruct model on port 30001. Evaluation uses GPT-4.1-mini.

Resources

Code SFT1 Checkpoint GRPO Checkpoint Dataset Benchmarks

The question I started with, where should compute go, has a messier answer than I expected. GRPO improved single-attempt success here, while four sampled attempts brought SFT1 close to the RL checkpoint. The deployment decision depends on the cost of those attempts and whether a real verifier can recover the success that Pass@4 exposes.