FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization
We present Future-KL Influenced Policy Optimization (FIPO), a reinforcement learning algorithm designed to overcome reasoning bottlenecks in large language models. While GRPO style training scales effectively, it typically relies on outcome-based rewards (ORM) that distribute a global advantage uniformly across every token in a trajectory. We argue that this coarse-grained credit assignment imposes a performance ceiling by failing to distinguish critical logical pivots from trivial tokens. FIPO addresses this by incorporating discounted future-KL divergence into the policy update, creating a dense advantage formulation that re-weights tokens based on their influence on subsequent trajectory behavior. Empirically, FIPO enables models to break through the length stagnation seen in standard baselines. Evaluated on Qwen2.5-32B, FIPO extends the average chain-of-thought length from roughly 4,000 to over 10,000 tokens and increases AIME 2024 Pass@1 accuracy from 50.0% to a peak of 58.0% (converging at approximately 56.0\%). This outperforms both DeepSeek-R1-Zero-Math-32B (around 47.0%) and o1-mini (approximately 56.0%). Our results suggest that establishing dense advantage formulations is a vital path for evolving ORM-based algorithms to unlock the full reasoning potential of base models. We open-source our training system, built on the verl framework.
69c1488ce9374d7b3df94d2f
Chiyu Ma
claimed_verified
2026-04-01T08:36:26.012Z
650df49b8f3228d807623533
/avatars/44e6646f17ef0b7e271d1a279b2359fd.svg
Chiyu Ma 
chiyum609
user
chiyum609
69c1488ce9374d7b3df94d30
Shuo Yang
claimed_verified
2026-04-01T08:36:35.577Z
656aa5dd3dc1d277e5214144
/avatars/32eb7d1f0a505fb853a1722f9ff655bd.svg
Shuo Yang
ShuoY
user
ShuoY
69c1488ce9374d7b3df94d31
Kexin Huang
claimed_verified
2026-03-30T09:10:47.384Z
6547a247dbce6bd2be168d33
/avatars/23ae384f7c4fc1a89a556a37f5e75acf.svg
Kexin Huang
737443h
user
737443h
69c1488ce9374d7b3df94d32
Jinda Lu
69c1488ce9374d7b3df94d33
Haoming Meng
69c1488ce9374d7b3df94d34
Shangshang Wang
claimed_verified
2026-03-25T10:51:27.023Z
67469d6a8407f929491dce06
https://cdn-avatars.huggingface.co/v1/production/uploads/67469d6a8407f929491dce06/7vd7zyApmT4rXe58gqmG1.png
Shangshang Wang
upup-ashton-wang
user
upup-ashton-wang
69c1488ce9374d7b3df94d35
Bolin Ding
69c1488ce9374d7b3df94d36
Soroush Vosoughi
claimed_verified
2026-04-15T12:09:23.472Z
648169d3cb64a2b85a4606e6
https://cdn-avatars.huggingface.co/v1/production/uploads/648169d3cb64a2b85a4606e6/6nEeE3TWvO9O5dCJTOG0C.jpeg
Soroush Vosoughi
soroushv
user
soroushv
69c1488ce9374d7b3df94d37
Guoyin Wang
69c1488ce9374d7b3df94d38
Jingren Zhou
FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization
We present Future-KL Influenced Policy Optimization (FIPO), a reinforcement learning algorithm designed to overcome reasoning bottlenecks in large language models. While GRPO style training scales effectively, it typically relies on outcome-based rewards (ORM) that distribute a global advantage uniformly across every token in a trajectory. We argue that this coarse-grained credit assignment imposes a performance ceiling by failing to distinguish critical logical pivots from trivial tokens. FIPO addresses this by incorporating discounted future-KL divergence into the policy update, creating a dense advantage formulation that re-weights tokens based on their influence on subsequent trajectory behavior. Empirically, FIPO enables models to break through the length stagnation seen in standard baselines. Evaluated on Qwen2.5-32B, FIPO extends the average chain-of-thought length from roughly 4,000 to over 10,000 tokens and increases AIME 2024 Pass@1 accuracy from 50.0% to a peak of 58.0% (converging at approximately 56.0\%). This outperforms both DeepSeek-R1-Zero-Math-32B (around 47.0%) and o1-mini (approximately 56.0%). Our results suggest that establishing dense advantage formulations is a vital path for evolving ORM-based algorithms to unlock the full reasoning potential of base models. We open-source our training system, built on the verl framework.
FIPO enhances reinforcement learning for language models by using discounted future-KL divergence to improve credit assignment and extend reasoning chains, achieving better mathematical problem-solving performance.
reinforcement learning policy optimization discounted future-KL divergence policy update dense advantage formulation chain-of-thought reasoning AIME 2024 Pass@1 accuracy