On the Optimal Reasoning Length for RL-Trained Language Models
Reinforcement learning substantially improves reasoning in large language models, but it also tends to lengthen chain of thought outputs and increase computational cost during both training and inference. Though length control methods have been proposed, it remains unclear what the optimal output length is for balancing efficiency and performance. In this work, we compare several length control methods on two models, Qwen3-1.7B Base and DeepSeek-R1-Distill-Qwen-1.5B. Our results indicate that length penalties may hinder reasoning acquisition, while properly tuned length control can improve efficiency for models with strong prior reasoning. By extending prior work to RL trained policies, we identify two failure modes, 1) long outputs increase dispersion, and 2) short outputs lead to under-thinking.
698c7521eb12ea7453916832
Daisuke Nohara
claimed_verified
2026-02-11T12:43:42.483Z
67f7ed6ae64695f24a724be1
https://cdn-avatars.huggingface.co/v1/production/uploads/67f7ed6ae64695f24a724be1/DTeiPbxey03v4kBmbJkaC.png
Daisuke Nohara
neodymium6
user
neodymium6
698c7521eb12ea7453916833
Taishi Nakamura
claimed_verified
2026-02-11T22:16:31.194Z
6308c49c454dc257521bc7f9
https://cdn-avatars.huggingface.co/v1/production/uploads/6308c49c454dc257521bc7f9/UWUS6OPa6OpVu1T0gd-wJ.jpeg
Taishi
Taishi-N324
user
Taishi-N324
698c7521eb12ea7453916834
Rio Yokota
On the Optimal Reasoning Length for RL-Trained Language Models
Reinforcement learning substantially improves reasoning in large language models, but it also tends to lengthen chain of thought outputs and increase computational cost during both training and inference. Though length control methods have been proposed, it remains unclear what the optimal output length is for balancing efficiency and performance. In this work, we compare several length control methods on two models, Qwen3-1.7B Base and DeepSeek-R1-Distill-Qwen-1.5B. Our results indicate that length penalties may hinder reasoning acquisition, while properly tuned length control can improve efficiency for models with strong prior reasoning. By extending prior work to RL trained policies, we identify two failure modes, 1) long outputs increase dispersion, and 2) short outputs lead to under-thinking.
Length control methods in reinforcement learning-trained language models affect reasoning performance and computational efficiency, with optimal output lengths balancing these factors.
reinforcement learning chain of thought length control methods reasoning acquisition output length computational cost efficiency performance