Defeating the Training-Inference Mismatch via FP16
Reinforcement learning (RL) fine-tuning of large language models (LLMs) often
suffers from instability due to the numerical mismatch between the training and
inference policies. While prior work has attempted to mitigate this issue
through algorithmic corrections or engineering alignments, we show that its
root cause lies in the floating point precision itself. The widely adopted
BF16, despite its large dynamic range, introduces large rounding errors that
breaks the consistency between training and inference. In this work, we
demonstrate that simply reverting to FP16 effectively eliminates this
mismatch. The change is simple, fully supported by modern frameworks with only
a few lines of code change, and requires no modification to the model
architecture or learning algorithm. Our results suggest that using FP16
uniformly yields more stable optimization, faster convergence, and stronger
performance across diverse tasks, algorithms and frameworks. We hope these
findings motivate a broader reconsideration of precision trade-offs in RL
fine-tuning.
6908202f812eca10f9cc5e89
Penghui Qi
claimed_verified
2025-11-03T20:52:13.465Z
63885f1d0bebb233d8ad6e5b
https://cdn-avatars.huggingface.co/v1/production/uploads/1669881620925-noauth.jpeg
Penghui Qi
QPHutu
user
QPHutu
6908202f812eca10f9cc5e8a
Zichen Liu
6908202f812eca10f9cc5e8b
Xiangxin Zhou
claimed_verified
2025-11-03T20:52:09.833Z
66129c7b50350afe76757262
/avatars/a2f4fac076b9d658a0d904ed54960f6f.svg
Xiangxin Zhou
zhouxiangxin
user
zhouxiangxin
6908202f812eca10f9cc5e8c
Tianyu Pang
6908202f812eca10f9cc5e8d
Chao Du
6908202f812eca10f9cc5e8e
Wee Sun Lee
6908202f812eca10f9cc5e8f
Min Lin
Defeating the Training-Inference Mismatch via FP16
Reinforcement learning (RL) fine-tuning of large language models (LLMs) often
suffers from instability due to the numerical mismatch between the training and
inference policies. While prior work has attempted to mitigate this issue
through algorithmic corrections or engineering alignments, we show that its
root cause lies in the floating point precision itself. The widely adopted
BF16, despite its large dynamic range, introduces large rounding errors that
breaks the consistency between training and inference. In this work, we
demonstrate that simply reverting to FP16 effectively eliminates this
mismatch. The change is simple, fully supported by modern frameworks with only
a few lines of code change, and requires no modification to the model
architecture or learning algorithm. Our results suggest that using FP16
uniformly yields more stable optimization, faster convergence, and stronger
performance across diverse tasks, algorithms and frameworks. We hope these
findings motivate a broader reconsideration of precision trade-offs in RL
fine-tuning.
Using FP16 precision in reinforcement learning fine-tuning of large language models improves stability, convergence, and performance by addressing numerical mismatches.
reinforcement learning large language models numerical mismatch training policies inference policies floating point precision BF16 FP16 stability convergence performance