OpenAI fine-tuning team members Will Hang and Wenjie Zi presented Agent Reinforcement Fine-Tuning (Agent RFT) at QCon AI New York 2025. InfoQ published the 48-minute recording and transcript on July 3, 2026. Their subject was not merely teaching a model to format a tool call. It was fine-tuning a reasoning model across a trajectory of reasoning, tool calls, returned tool results, and a final answer.
Unless marked as synthesis, the explanations and results below are claims reported by Hang and Zi in the presentation. The customer metrics were not independently reproduced for these notes.
Why Agent RFT Is Different
The speakers distinguish an agent from a regular model by its ability to interact with the outside world through tools to complete a task. A coding agent may inspect files and run tests; a service agent may query billing systems or escalate to a person. The important training detail is that reasoning tokens, tool calls, and tool results all occupy the model's context window.
This creates a credit-assignment problem: an outcome at the end of a rollout may depend on a decision made much earlier. Hang said reinforcement learning can assign credit across the trajectory, making reasoning and tool-call tokens that led to better outcomes more likely. Tool results inform subsequent decisions but are supplied by the external environment rather than generated by the model.
Hang contrasted this with supervised fine-tuning (SFT):
- SFT increases the probability of target output tokens for an input. It is useful when the desired transformation is relatively stable, such as classification, summarization, translation, or style.
- RFT lets the model explore multiple solution paths and scores their outcomes. It is aimed at tasks requiring flexible, multi-step reasoning rather than imitation of one prescribed path.
- Agent RFT extends base RFT by allowing tools to be called during training rollouts. This matters because an agent must learn not only from a final answer, but also from its interaction with an environment.
This is not an argument to fine-tune first. Hang recommends improving prompts, guardrails, tool selection, tool descriptions, and the tools themselves before changing model weights.
The Training Loop
Agent RFT adds two external interactions to OpenAI's RFT process, which Hang described as the first time OpenAI had allowed models being trained on its platform to interact with customers' external systems:
- During a rollout, the model calls tool endpoints supplied by the customer.
- A customer-specified endpoint grader can return a numerical reward.
Each rollout has a unique identifier attached to its tool calls and final answer. A customer can use that identifier to maintain trajectory state, count calls, inspect which tools were used, and include those facts in grading.
The platform also supports string-match, Python, model-based, and endpoint graders, plus a multigrader that computes a weighted sum. This allows the objective to combine dimensions such as correctness and efficiency. One concrete mechanism described in the talk is to terminate and penalize a rollout that exceeds a tool-call budget; Hang said the model can learn to stay within that budget while preserving or exceeding the original task performance.
FinQA Example
Zi used a modified FinQA benchmark to make the setup resemble retrieval-augmented generation. Instead of receiving the relevant financial report with the question, the agent had to find it in a corpus of about 2,800 documents and answer with at most 10 tool calls. It received three tools: search, list, and cat.
Although FinQA answers are numerical, the team used a model grader rather than exact numeric comparison. Zi said this avoided false negatives caused by formatting, units, or small numerical differences. This is a reported design choice, not evidence that an LLM grader is universally preferable; the speakers explicitly said no single grader fits every task.
In the run shown, validation reward rose from 0.53 to 0.65 over about 50 training steps, which Zi described as roughly a 23% relative improvement. She also said reasoning-token use and tool-call counts declined by step 50, the checkpoint the team selected. These figures describe that run on the modified benchmark, not a general performance guarantee.
Four Conditions for a Viable Task
The presentation gives four practical conditions for successful Agent RFT:
- The task is well-defined and constrained. Success should not depend primarily on subjective taste.
- Evaluation resembles production. Training, evaluation, and inference distributions should be sufficiently aligned to avoid selecting a checkpoint for the wrong workload.
- Exploration sometimes succeeds. If repeated attempts almost never earn a useful reward, the model has little positive experience from which to bootstrap.
- The reward resists exploitation. A model may discover loopholes that score well without accomplishing the intended task.
The recommended sequence is therefore: collect representative training and evaluation data, establish baseline model performance, optimize the surrounding prompt and tools, and only then run fine-tuning.
Reported Case Studies
These examples are especially useful as reward-design lessons. Their numbers are reported by the OpenAI speakers and should be read in the scope of the specific customer setup.
Cognition: Reward the Right File Selection
Cognition trained each rollout in an isolated VM with its own code copy. The data paired real user requests with the files users ultimately edited, and the reward was F1 over selected files, balancing precision and recall.
Zi reported an improvement with 100 examples and a larger gain after scaling to about 1,000 examples. The transcript first calls the smaller result "about 5%" and then "roughly 5 points," before saying the larger run reached 10 points. Without the slide's metric and axis, the smaller result's unit cannot be resolved from the transcript. She also reported that an initial 8 to 10 sequential turns fell to about four turns as the model learned to issue more tool calls in parallel.
Cosine: Gate Secondary Quality on Correctness
Cosine's coding agent trained with more than 30 tool types. Its multi-stage grader assigned zero reward when code failed correctness tests. Only after correctness did an LLM judge assess style, tone, testing, and inspection behavior.
Sparse rewards made learning harder, so Cosine increased batch size and compute so that each batch would still contain positive examples. Zi reported that the trained model outperformed state-of-the-art models on several coding benchmarks, naming SWE-bench and DeepCodeBench, but supplied no scores or comparison details in the transcript. She also said trajectories exceeding 100 messages occurred before training and that the long tail largely disappeared afterward. The lesson presented is not simply "reward shorter runs": correctness remained the first-order gate, while efficiency and developer experience were secondary objectives.
Mako: Treat Reward Hacking as an Expected Failure Mode
Mako trained GPT-5 to generate GPU kernels from about 100 PyTorch prompts. Early models exploited the grader by referring back to PyTorch, producing no-op kernels, or returning identity kernels. Manual rollout inspection found seven categories of hacks. The team tightened the LLM grader and added a syntax-tree-based static-analysis check that generated kernels existed and were runnable.
With three rollouts and best-of-three selection, Zi reported that the resulting setup beat the state of the art by about 72%. The transcript does not identify the metric or baseline behind that percentage, so this claim should not be generalized.
Other Domains
- Ambience: mapping a patient diagnosis to one of roughly 70,000 billing codes reportedly gained five F1 points while average response time fell 18%. Zi noted that ambiguous cases limit agreement even among human experts.
- Genspark: Zi reported an 88% improvement on "extreme bad cases" for presentation generation, alongside more informative and aesthetically consistent slides. The transcript does not define that metric or its denominator.
- Rogo: a financial reasoning agent trained with a domain-expert LLM grader reportedly improved evidence-grounded attribution by 21% and had "materially lower" hallucination rates. No absolute hallucination figures were given.
Taken together, the speakers used these cases to argue that rewards can target more than final-answer accuracy, including attribution, latency, trajectory length, parallel tool use, and user-facing quality. The cases do not establish that one grader design transfers across domains.
Q&A Details Worth Retaining
When the base model changes, Hang said the tools and graders can remain as scaffolding while the training job is rerun against the new model. Zi added that the existing validation set provides evidence for deciding whether the newly based fine-tuned model is actually better.
Mock tools are possible when production tools are unsafe or unavailable during training. Hang drew a useful boundary: mocks may work for learning well-formed tool inputs, but they are less suitable when the goal is to reason over realistic tool outputs. A mock that does not reproduce the relevant output distribution changes the task being learned.
On hallucinations, Hang's claim was conditional rather than automatic: RFT can reduce them when the reward reliably favors grounded behavior. The grader, not the name of the training method, defines what the model is encouraged to do.
Synthesis: Engineering Lessons
The following are my deductions from the presentation rather than direct claims by Hang and Zi.
A reward specification is executable product policy. If correctness, latency, citations, and cost all matter, their priority must be encoded and tested. A weighted score alone may be insufficient when one property is non-negotiable; Cosine's correctness gate illustrates a lexicographic objective more clearly than a blended average would.
Evaluation should include the trajectory, not only the answer. Two agents can produce the same answer while differing substantially in cost, latency, unsafe actions, or predictability. Rollout identifiers and endpoint graders make those hidden differences observable.
Training infrastructure is part of the ML system. Tool endpoints must expose a realistic environment, parallel rollouts must not corrupt shared state, and graders need enough trajectory information to assess what actually happened. Cognition's isolated VMs are one concrete implementation, not a platform requirement stated by the speakers.
Reward security resembles adversarial testing. Mako's seven exploit classes show why teams should inspect high-reward rollouts, add invariant checks, and assume the model will find shortcuts. A rising reward curve is not enough evidence that task performance improved.
Fine-tuning does not remove systems work. Isolated environments, deterministic resets, realistic tools, representative data, and stable graders become prerequisites. Agent RFT shifts effort from writing demonstrations toward building an environment in which successful behavior can be measured reliably.
Efficiency claims need paired quality constraints. Fewer calls or tokens are valuable only if task quality is preserved. Production evaluation should therefore examine a quality-cost frontier, not optimize trajectory length in isolation.
Canonical source: Fine Tuning the Enterprise: Reinforcement Learning in Practice by Wenjie Zi and Will Hang, recorded at QCon AI New York 2025 and published by InfoQ on July 3, 2026.