Online Causal Kalman Filtering for Stable and Effective Policy Optimization
Reinforcement learning for large language models suffers from high-variance token-level importance sampling (IS) ratios, which would destabilize policy optimization at scale. To improve stability, recent methods typically use a fixed sequence-level IS ratio for all tokens in a sequence or adjust each token's IS ratio separately, thereby neglecting temporal off-policy derivation across tokens in a sequence. In this paper, we first empirically identify that local off-policy deviation is structurally inconsistent at the token level, which may distort policy-gradient updates across adjacent tokens and lead to training collapse. To address the issue, we propose Online Causal Kalman Filtering for stable and effective Policy Optimization (KPO). Concretely, we model the desired IS ratio as a latent state that evolves across tokens and apply a Kalman filter to update this state online and autoregressively based on the states of past tokens, regardless of future tokens. The resulting filtered IS ratios preserve token-wise local structure-aware variation while strongly smoothing noise spikes, yielding more stable and effective policy updates. Experimentally, KPO achieves superior results on challenging math reasoning datasets compared with state-of-the-art counterparts.
698d37a66c5152984e4f3e7b
Shuo He
698d37a66c5152984e4f3e7c
Lang Feng
claimed_verified
2026-02-12T20:23:25.095Z
66ba29dd59e8e7a957154c5f
https://cdn-avatars.huggingface.co/v1/production/uploads/66ba29dd59e8e7a957154c5f/VvVS7IZNPUIB023GAEf5u.png
Lang Feng
langfeng01
user
langfeng01
698d37a66c5152984e4f3e7d
Xin Cheng
698d37a66c5152984e4f3e7e
Lei Feng
claimed_verified
2026-02-12T13:57:10.067Z
698dc46341d49f1062061c64
/avatars/34d7f69e61aeb37e226da8488f8b8c98.svg
Lei Feng
CharlesFeng1995
user
CharlesFeng1995
698d37a66c5152984e4f3e7f
Bo An
Online Causal Kalman Filtering for Stable and Effective Policy Optimization
Reinforcement learning for large language models suffers from high-variance token-level importance sampling (IS) ratios, which would destabilize policy optimization at scale. To improve stability, recent methods typically use a fixed sequence-level IS ratio for all tokens in a sequence or adjust each token's IS ratio separately, thereby neglecting temporal off-policy derivation across tokens in a sequence. In this paper, we first empirically identify that local off-policy deviation is structurally inconsistent at the token level, which may distort policy-gradient updates across adjacent tokens and lead to training collapse. To address the issue, we propose Online Causal Kalman Filtering for stable and effective Policy Optimization (KPO). Concretely, we model the desired IS ratio as a latent state that evolves across tokens and apply a Kalman filter to update this state online and autoregressively based on the states of past tokens, regardless of future tokens. The resulting filtered IS ratios preserve token-wise local structure-aware variation while strongly smoothing noise spikes, yielding more stable and effective policy updates. Experimentally, KPO achieves superior results on challenging math reasoning datasets compared with state-of-the-art counterparts.
Online Causal Kalman Filtering addresses high-variance token-level importance sampling in reinforcement learning for large language models by modeling IS ratios as evolving latent states and using Kalman filtering for stable policy optimization.
importance sampling policy optimization reinforcement learning large language models Kalman filter token-level sequence-level off-policy derivation policy gradient training collapse