Supervised Reinforcement Learning: From Expert Trajectories to Step-wise
  Reasoning
Large Language Models (LLMs) often struggle with problems that require
multi-step reasoning. For small-scale open-source models, Reinforcement
Learning with Verifiable Rewards (RLVR) fails when correct solutions are rarely
sampled even after many attempts, while Supervised Fine-Tuning (SFT) tends to
overfit long demonstrations through rigid token-by-token imitation. To address
this gap, we propose Supervised Reinforcement Learning (SRL), a framework that
reformulates problem solving as generating a sequence of logical "actions". SRL
trains the model to generate an internal reasoning monologue before committing
to each action. It provides smoother rewards based on the similarity between
the model's actions and expert actions extracted from the SFT dataset in a
step-wise manner. This supervision offers richer learning signals even when all
rollouts are incorrect, while encouraging flexible reasoning guided by expert
demonstrations. As a result, SRL enables small models to learn challenging
problems previously unlearnable by SFT or RLVR. Moreover, initializing training
with SRL before refining with RLVR yields the strongest overall performance.
Beyond reasoning benchmarks, SRL generalizes effectively to agentic software
engineering tasks, establishing it as a robust and versatile training framework
for reasoning-oriented LLMs.
6904169e2c556835fa67f004
Yihe Deng
6904169e2c556835fa67f005
I-Hung Hsu
6904169e2c556835fa67f006
Jun Yan
6904169e2c556835fa67f007
Zifeng Wang
6904169e2c556835fa67f008
Rujun Han
6904169e2c556835fa67f009
Gufeng Zhang
6904169e2c556835fa67f00a
Yanfei Chen
6904169e2c556835fa67f00b
Wei Wang
6904169e2c556835fa67f00c
Tomas Pfister
6904169e2c556835fa67f00d
Chen-Yu Lee
Supervised Reinforcement Learning: From Expert Trajectories to Step-wise
  Reasoning
Large Language Models (LLMs) often struggle with problems that require
multi-step reasoning. For small-scale open-source models, Reinforcement
Learning with Verifiable Rewards (RLVR) fails when correct solutions are rarely
sampled even after many attempts, while Supervised Fine-Tuning (SFT) tends to
overfit long demonstrations through rigid token-by-token imitation. To address
this gap, we propose Supervised Reinforcement Learning (SRL), a framework that
reformulates problem solving as generating a sequence of logical "actions". SRL
trains the model to generate an internal reasoning monologue before committing
to each action. It provides smoother rewards based on the similarity between
the model's actions and expert actions extracted from the SFT dataset in a
step-wise manner. This supervision offers richer learning signals even when all
rollouts are incorrect, while encouraging flexible reasoning guided by expert
demonstrations. As a result, SRL enables small models to learn challenging
problems previously unlearnable by SFT or RLVR. Moreover, initializing training
with SRL before refining with RLVR yields the strongest overall performance.
Beyond reasoning benchmarks, SRL generalizes effectively to agentic software
engineering tasks, establishing it as a robust and versatile training framework
for reasoning-oriented LLMs.
Supervised Reinforcement Learning (SRL) enhances small-scale LLMs' multi-step reasoning by generating internal monologues and using step-wise expert actions for richer learning signals, outperforming SFT and RLVR.
Reinforcement Learning with Verifiable Rewards Supervised Fine-Tuning Supervised Reinforcement Learning internal reasoning monologue expert actions step-wise learning signals agentic software engineering tasks