Rethinking Expert Trajectory Utilization in LLM Post-training
While effective post-training integrates Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), the optimal mechanism for utilizing expert trajectories remains unresolved. We propose the Plasticity-Ceiling Framework to theoretically ground this landscape, decomposing performance into foundational SFT performance and the subsequent RL plasticity. Through extensive benchmarking, we establish the Sequential SFT-then-RL pipeline as the superior standard, overcoming the stability deficits of synchronized approaches. Furthermore, we derive precise scaling guidelines: (1) Transitioning to RL at the SFT Stable or Mild Overfitting Sub-phase maximizes the final ceiling by securing foundational SFT performance without compromising RL plasticity; (2) Refuting ``Less is More'' in the context of SFT-then-RL scaling, we demonstrate that Data Scale determines the primary post-training potential, while Trajectory Difficulty acts as a performance multiplier; and (3) Identifying that the Minimum SFT Validation Loss serves as a robust indicator for selecting the expert trajectories that maximize the final performance ceiling. Our findings provide actionable guidelines for maximizing the value extracted from expert trajectories.
69417c3e65f1e24a117808e8
Bowen Ding
69417c3e65f1e24a117808e9
Yuhan Chen
69417c3e65f1e24a117808ea
Jiayang Lv
69417c3e65f1e24a117808eb
Jiyao Yuan
69417c3e65f1e24a117808ec
Qi Zhu
69417c3e65f1e24a117808ed
Shuangshuang Tian
69417c3e65f1e24a117808ee
Dantong Zhu
claimed_verified
2026-02-09T08:37:30.279Z
67eb81928ebaba352ed56c3b
https://cdn-avatars.huggingface.co/v1/production/uploads/67eb81928ebaba352ed56c3b/kTV3pDz5UEc9NJ80Gz9_N.jpeg
dt
zhudantong
user
zhudantong
69417c3e65f1e24a117808ef
Futing Wang
69417c3e65f1e24a117808f0
Heyuan Deng
69417c3e65f1e24a117808f1
Fei Mi
69417c3e65f1e24a117808f2
Lifeng Shang
69417c3e65f1e24a117808f3
Tao Lin
Rethinking Expert Trajectory Utilization in LLM Post-training
While effective post-training integrates Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), the optimal mechanism for utilizing expert trajectories remains unresolved. We propose the Plasticity-Ceiling Framework to theoretically ground this landscape, decomposing performance into foundational SFT performance and the subsequent RL plasticity. Through extensive benchmarking, we establish the Sequential SFT-then-RL pipeline as the superior standard, overcoming the stability deficits of synchronized approaches. Furthermore, we derive precise scaling guidelines: (1) Transitioning to RL at the SFT Stable or Mild Overfitting Sub-phase maximizes the final ceiling by securing foundational SFT performance without compromising RL plasticity; (2) Refuting ``Less is More'' in the context of SFT-then-RL scaling, we demonstrate that Data Scale determines the primary post-training potential, while Trajectory Difficulty acts as a performance multiplier; and (3) Identifying that the Minimum SFT Validation Loss serves as a robust indicator for selecting the expert trajectories that maximize the final performance ceiling. Our findings provide actionable guidelines for maximizing the value extracted from expert trajectories.
The Sequential SFT-then-RL pipeline is identified as optimal for integrating expert trajectories, with guidelines for scaling and trajectory selection based on performance metrics.
Supervised Fine-Tuning Reinforcement Learning Plasticity-Ceiling Framework Sequential SFT-then-RL SFT Stable Mild Overfitting Sub-phase Data Scale Trajectory Difficulty Minimum SFT Validation Loss