Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive
  Exploration for Agentic Reinforcement Learning
Reinforcement learning (RL) is the dominant paradigm for sharpening strategic
tool use capabilities of LLMs on long-horizon, sparsely-rewarded agent tasks,
yet it faces a fundamental challenge of exploration-exploitation trade-off.
Existing studies stimulate exploration through the lens of policy entropy, but
such mechanical entropy maximization is prone to RL training instability due to
the multi-turn distribution shifting. In this paper, we target the progressive
exploration-exploitation balance under the guidance of the agent own
experiences without succumbing to either entropy collapsing or runaway
divergence. We propose SPEAR, a curriculum-based self-imitation learning (SIL)
recipe for training agentic LLMs. It extends the vanilla SIL framework, where a
replay buffer stores self-generated promising trajectories for off-policy
update, by gradually steering the policy evolution within a well-balanced range
of entropy across stages. Specifically, our approach incorporates a curriculum
to manage the exploration process, utilizing intrinsic rewards to foster
skill-level exploration and facilitating action-level exploration through SIL.
At first, the auxiliary tool call reward plays a critical role in the
accumulation of tool-use skills, enabling broad exposure to the unfamiliar
distributions of the environment feedback with an upward entropy trend. As
training progresses, self-imitation gets strengthened to exploit existing
successful patterns from replayed experiences for comparative action-level
exploration, accelerating solution iteration without unbounded entropy growth.
To further stabilize training, we recalibrate the advantages of experiences in
the replay buffer to address the potential policy drift. Reugularizations such
as the clipping of tokens with high covariance between probability and
advantage are introduced to the trajectory-level entropy control to curb
over-confidence.
68d9ee7d0177a6054b013abb
Yulei Qin
claimed_verified
2025-10-10T09:58:40.812Z
6390525c00fb8ec4a424e0ff
/avatars/4302571e2ef4a9875491221aa630a329.svg
Yulei Qin
yolay
user
yolay
68d9ee7d0177a6054b013abc
Xiaoyu Tan
claimed_verified
2026-01-05T08:47:00.586Z
637af0a7bdf7309aa6da1c36
https://cdn-avatars.huggingface.co/v1/production/uploads/637af0a7bdf7309aa6da1c36/NHZ-09otVCfbpXVxm8f-e.png
Xiaoyu Tan
WIlliam1900
user
WIlliam1900
68d9ee7d0177a6054b013abd
Zhengbao He
68d9ee7d0177a6054b013abe
Gang Li
68d9ee7d0177a6054b013abf
Haojia Lin
68d9ee7d0177a6054b013ac0
Zongyi Li
68d9ee7d0177a6054b013ac1
Zihan Xu
68d9ee7d0177a6054b013ac2
Yuchen Shi
claimed_verified
2026-01-04T20:32:43.944Z
622b00a776c20fee5d14501b
/avatars/e00496dda1e309548e7b5b437839bb65.svg
Eason shi
Easonshi
user
Easonshi
68d9ee7d0177a6054b013ac3
Siqi Cai
68d9ee7d0177a6054b013ac4
Renting Rui
68d9ee7d0177a6054b013ac5
Shaofei Cai
claimed_verified
2025-12-31T20:58:50.230Z
6578459d62d3ac1817ed79fe
https://cdn-avatars.huggingface.co/v1/production/uploads/6578459d62d3ac1817ed79fe/AXDJuwLUoEOb4Fj3U0Xxo.jpeg
Shaofei Cai
phython96
user
phython96
68d9ee7d0177a6054b013ac6
Yuzheng Cai
claimed_verified
2025-10-10T09:58:32.418Z
66e258bdc70c02b46dfed6e3
/avatars/ccc2d604616c018f45a268a610472cac.svg
Yuzheng Cai
Ucreate
user
Ucreate
68d9ee7d0177a6054b013ac7
Xuan Zhang
68d9ee7d0177a6054b013ac8
Sheng Ye
68d9ee7d0177a6054b013ac9
Ke Li
claimed_verified
2026-01-06T10:06:44.192Z
63280915eeee4dd858083092
/avatars/78347af4af42527d53e88d9969c5c934.svg
Ke Li
tristanli
user
tristanli
68d9ee7d0177a6054b013aca
Xing Sun
claimed_verified
2025-09-29T11:26:42.729Z
647401e50da364bd0d002f2a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/vPuPn7EV092mLBOM2YZXd.png
XING SUN
tedsun
user
tedsun
Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive
  Exploration for Agentic Reinforcement Learning
Reinforcement learning (RL) is the dominant paradigm for sharpening strategic
tool use capabilities of LLMs on long-horizon, sparsely-rewarded agent tasks,
yet it faces a fundamental challenge of exploration-exploitation trade-off.
Existing studies stimulate exploration through the lens of policy entropy, but
such mechanical entropy maximization is prone to RL training instability due to
the multi-turn distribution shifting. In this paper, we target the progressive
exploration-exploitation balance under the guidance of the agent own
experiences without succumbing to either entropy collapsing or runaway
divergence. We propose SPEAR, a curriculum-based self-imitation learning (SIL)
recipe for training agentic LLMs. It extends the vanilla SIL framework, where a
replay buffer stores self-generated promising trajectories for off-policy
update, by gradually steering the policy evolution within a well-balanced range
of entropy across stages. Specifically, our approach incorporates a curriculum
to manage the exploration process, utilizing intrinsic rewards to foster
skill-level exploration and facilitating action-level exploration through SIL.
At first, the auxiliary tool call reward plays a critical role in the
accumulation of tool-use skills, enabling broad exposure to the unfamiliar
distributions of the environment feedback with an upward entropy trend. As
training progresses, self-imitation gets strengthened to exploit existing
successful patterns from replayed experiences for comparative action-level
exploration, accelerating solution iteration without unbounded entropy growth.
To further stabilize training, we recalibrate the advantages of experiences in
the replay buffer to address the potential policy drift. Reugularizations such
as the clipping of tokens with high covariance between probability and
advantage are introduced to the trajectory-level entropy control to curb
over-confidence.
SPEAR, a curriculum-based self-imitation learning method, manages exploration-exploitation balance in reinforcement learning for LLMs by using intrinsic rewards and trajectory-level entropy control.
reinforcement learning exploration-exploitation trade-off policy entropy curriculum-based self-imitation learning replay buffer off-policy update intrinsic rewards self-imitation entropy control policy drift token clipping