Think Longer to Explore Deeper: Learn to Explore In-Context via Length-Incentivized Reinforcement Learning
Achieving effective test-time scaling requires models to engage in In-Context Exploration -- the intrinsic ability to generate, verify, and refine multiple reasoning hypotheses within a single continuous context.
  Grounded in State Coverage theory, our analysis identifies a critical bottleneck to enabling this capability: while broader state coverage requires longer reasoning trajectories, the probability of sampling such sequences decays exponentially during autoregressive generation, a phenomenon we term the ``Shallow Exploration Trap''.
  To bridge this gap, we propose Length-Incentivized Exploration(\method).
  This simple yet effective recipe explicitly encourages models to explore more via a length-based reward coupled with a redundancy penalty, thereby maximizing state coverage in two-step manner.
  Comprehensive experiments across different models (Qwen3, Llama) demonstrate that \method effectively incentivize in-context exploration.
  As a result, our method achieves an average improvement of 4.4\% on in-domain tasks and a 2.7\% gain on out-of-domain benchmarks.
698ec188cace060ff123aed7
Futing Wang
698ec188cace060ff123aed8
Jianhao Yan
698ec188cace060ff123aed9
Yun Luo
698ec188cace060ff123aeda
Ganqu Cui
698ec188cace060ff123aedb
Zhi Wang
698ec188cace060ff123aedc
Xiaoye Qu
698ec188cace060ff123aedd
Yue Zhang
698ec188cace060ff123aede
Yu Cheng
698ec188cace060ff123aedf
Tao Lin
Think Longer to Explore Deeper: Learn to Explore In-Context via Length-Incentivized Reinforcement Learning
Achieving effective test-time scaling requires models to engage in In-Context Exploration -- the intrinsic ability to generate, verify, and refine multiple reasoning hypotheses within a single continuous context.
  Grounded in State Coverage theory, our analysis identifies a critical bottleneck to enabling this capability: while broader state coverage requires longer reasoning trajectories, the probability of sampling such sequences decays exponentially during autoregressive generation, a phenomenon we term the ``Shallow Exploration Trap''.
  To bridge this gap, we propose Length-Incentivized Exploration(\method).
  This simple yet effective recipe explicitly encourages models to explore more via a length-based reward coupled with a redundancy penalty, thereby maximizing state coverage in two-step manner.
  Comprehensive experiments across different models (Qwen3, Llama) demonstrate that \method effectively incentivize in-context exploration.
  As a result, our method achieves an average improvement of 4.4\% on in-domain tasks and a 2.7\% gain on out-of-domain benchmarks.
Models require in-context exploration capabilities to scale effectively at test time, but autoregressive generation faces exponential decay in sampling long sequences, which is addressed by a length-incentivized exploration method that improves performance on both in-domain and out-of-domain tasks.
In-Context Exploration State Coverage theory Shallow Exploration Trap Length-Incentivized Exploration autoregressive generation redundancy penalty state coverage