PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost
Post-training for long-horizon agentic tasks has a tension between compute efficiency and generalization. While supervised fine-tuning (SFT) is compute efficient, it often suffers from out-of-domain (OOD) degradation. Conversely, end-to-end reinforcement learning (E2E RL) preserves OOD capabilities, but incurs high compute costs due to many turns of on-policy rollout. We introduce PivotRL, a novel framework that operates on existing SFT trajectories to combine the compute efficiency of SFT with the OOD accuracy of E2E RL. PivotRL relies on two key mechanisms: first, it executes local, on-policy rollouts and filters for pivots: informative intermediate turns where sampled actions exhibit high variance in outcomes; second, it utilizes rewards for functional-equivalent actions rather than demanding strict string matching with the SFT data demonstration. We theoretically show that these mechanisms incentivize strong learning signals with high natural gradient norm, while maximally preserving policy probability ordering on actions unrelated to training tasks. In comparison to standard SFT on identical data, we demonstrate that PivotRL achieves +4.17% higher in-domain accuracy on average across four agentic domains, and +10.04% higher OOD accuracy in non-agentic tasks. Notably, on agentic coding tasks, PivotRL achieves competitive accuracy with E2E RL with 4x fewer rollout turns. PivotRL is adopted by NVIDIA's Nemotron-3-Super-120B-A12B, acting as the workhorse in production-scale agentic post-training.
69c20470c105f45cb0481eb8
Junkeun Yi
claimed_verified
2026-03-25T10:51:20.342Z
6865a40310765e1e6e954ddf
https://cdn-avatars.huggingface.co/v1/production/uploads/6865a40310765e1e6e954ddf/cjo98aPZ7sO7BgQokkwEn.jpeg
Junkeun Yi
jkyi-nvidia
user
jkyi-nvidia
69c20470c105f45cb0481eb9
Damon Mosk-Aoyama
69c20470c105f45cb0481eba
Baihe Huang
69c20470c105f45cb0481ebb
Ritu Gala
69c20470c105f45cb0481ebc
Charles Wang
69c20470c105f45cb0481ebd
Sugam Dipak Devare
69c20470c105f45cb0481ebe
Khushi Bhardwaj
69c20470c105f45cb0481ebf
Abhibha Gupta
69c20470c105f45cb0481ec0
Oleksii Kuchaiev
69c20470c105f45cb0481ec1
Jiantao Jiao
69c20470c105f45cb0481ec2
Jian Zhang
69c20470c105f45cb0481ec3
Venkat Srinivasan
PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost
Post-training for long-horizon agentic tasks has a tension between compute efficiency and generalization. While supervised fine-tuning (SFT) is compute efficient, it often suffers from out-of-domain (OOD) degradation. Conversely, end-to-end reinforcement learning (E2E RL) preserves OOD capabilities, but incurs high compute costs due to many turns of on-policy rollout. We introduce PivotRL, a novel framework that operates on existing SFT trajectories to combine the compute efficiency of SFT with the OOD accuracy of E2E RL. PivotRL relies on two key mechanisms: first, it executes local, on-policy rollouts and filters for pivots: informative intermediate turns where sampled actions exhibit high variance in outcomes; second, it utilizes rewards for functional-equivalent actions rather than demanding strict string matching with the SFT data demonstration. We theoretically show that these mechanisms incentivize strong learning signals with high natural gradient norm, while maximally preserving policy probability ordering on actions unrelated to training tasks. In comparison to standard SFT on identical data, we demonstrate that PivotRL achieves +4.17% higher in-domain accuracy on average across four agentic domains, and +10.04% higher OOD accuracy in non-agentic tasks. Notably, on agentic coding tasks, PivotRL achieves competitive accuracy with E2E RL with 4x fewer rollout turns. PivotRL is adopted by NVIDIA's Nemotron-3-Super-120B-A12B, acting as the workhorse in production-scale agentic post-training.
PivotRL is a novel framework that combines supervised fine-tuning efficiency with reinforcement learning generalization by using local rollouts and functional-equivalent action rewards to achieve better performance with reduced compute costs.
supervised fine-tuning reinforcement learning on-policy rollout policy probability ordering natural gradient norm action variance functional-equivalent actions agentic tasks post-training