Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning
Post-training of reasoning LLMs is a holistic process that typically consists of an offline SFT stage followed by an online reinforcement learning (RL) stage. However, SFT is often optimized in isolation to maximize SFT performance alone.
  We show that, after identical RL training, models initialized from stronger SFT checkpoints can significantly underperform those initialized from weaker ones. We attribute this to a mismatch typical in current SFT-RL pipelines: the distribution that generates the offline SFT data can differ substantially from the policy optimized during online RL, which learns from its own rollouts.
  We propose PEAR (Policy Evaluation-inspired Algorithm for Offline Learning Loss Re-weighting), an SFT-stage method that corrects this mismatch and better prepares the model for RL. PEAR uses importance sampling to reweight the SFT loss, with three variants operating at the token, block, and sequence levels. It can be used to augment standard SFT objectives and incurs little additional training overhead once probabilities for the offline data are collected.
  We conduct controlled experiments on verifiable reasoning games and mathematical reasoning tasks on Qwen 2.5 and 3 and DeepSeek-distilled models. PEAR consistently improves post-RL performance over canonical SFT, with pass at 8 gains up to a 14.6 percent on AIME2025. Our results suggest that PEAR is an effective step toward more holistic LLM post-training by designing and evaluating SFT with downstream RL in mind rather than in isolation.
69816dd5ce18b18628095fc9
Dylan Zhang
claimed_verified
2026-02-03T10:07:49.981Z
6705f27bbaae2d7ee2b4bf30
/avatars/ab79b73ca39ebc1793a2a8540d2ff1c6.svg
Zhang
Diluner
user
Diluner
69816dd5ce18b18628095fca
Yufeng Xu
69816dd5ce18b18628095fcb
Haojin Wang
claimed_verified
2026-02-25T17:32:57.721Z
645cd322a03f3ebb0bdecda9
/avatars/60efe7444373772a25f44ac1f0a2a507.svg
SeanWang0027
SeanWang0027
user
SeanWang0027
69816dd5ce18b18628095fcc
Qingzhi Chen
69816dd5ce18b18628095fcd
Hao Peng
Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning
Post-training of reasoning LLMs is a holistic process that typically consists of an offline SFT stage followed by an online reinforcement learning (RL) stage. However, SFT is often optimized in isolation to maximize SFT performance alone.
  We show that, after identical RL training, models initialized from stronger SFT checkpoints can significantly underperform those initialized from weaker ones. We attribute this to a mismatch typical in current SFT-RL pipelines: the distribution that generates the offline SFT data can differ substantially from the policy optimized during online RL, which learns from its own rollouts.
  We propose PEAR (Policy Evaluation-inspired Algorithm for Offline Learning Loss Re-weighting), an SFT-stage method that corrects this mismatch and better prepares the model for RL. PEAR uses importance sampling to reweight the SFT loss, with three variants operating at the token, block, and sequence levels. It can be used to augment standard SFT objectives and incurs little additional training overhead once probabilities for the offline data are collected.
  We conduct controlled experiments on verifiable reasoning games and mathematical reasoning tasks on Qwen 2.5 and 3 and DeepSeek-distilled models. PEAR consistently improves post-RL performance over canonical SFT, with pass at 8 gains up to a 14.6 percent on AIME2025. Our results suggest that PEAR is an effective step toward more holistic LLM post-training by designing and evaluating SFT with downstream RL in mind rather than in isolation.
Post-training of reasoning large language models can be improved by correcting distribution mismatches between supervised fine-tuning and reinforcement learning stages through importance sampling reweighting of the SFT loss.
supervised fine-tuning reinforcement learning policy evaluation importance sampling loss re-weighting post-training distribution mismatch verifiable reasoning mathematical reasoning Qwen DeepSeek