LoopRPT: Reinforcement Pre-Training for Looped Language Models
Looped language models (LoopLMs) perform iterative latent computation to refine internal representations, offering a promising alternative to explicit chain-of-thought (CoT) reasoning. However, existing reinforcement learning (RL) paradigms primarily target output tokens, creating a structural mismatch with looped architectures whose reasoning unfolds implicitly. In this work, we propose LoopRPT, a reinforcement pre-training framework tailored for LoopLMs. By reframing next-token prediction as a next-token reasoning task, LoopRPT assigns reinforcement signals directly to latent steps using an EMA teacher reference and noisy latent rollouts. This formulation enables RL to directly shape intermediate representations, compressing effective reasoning into fewer iterations. We instantiate LoopRPT on the Ouro architecture across multiple model scales. Results demonstrate that LoopRPT consistently improves per-step representation quality, achieving Pareto dominance in accuracy-computation trade-offs. Notably, significant gains on hard tokens indicate that LoopRPT enhances early-stage reasoning rather than merely encouraging premature exits. Our findings highlight reinforcement pre-training as a principled paradigm for learning efficient latent reasoning in LoopLMs.
69c0f5283fd3fef6237a6933
Guo Tang
claimed_verified
2026-03-23T09:15:29.450Z
667b1356f543b957eccf1fce
/avatars/5b396edf4604b7dc838bfc0b8bb3c667.svg
Tang Guo
JENLISA4EVER
user
JENLISA4EVER
69c0f5283fd3fef6237a6934
Shixin Jiang
claimed_verified
2026-03-23T09:15:31.973Z
64c36c2ec3633e5b923e87d3
/avatars/94fccc1b9eff68939c7bbcd011d6417f.svg
蒋世鑫
ThreeGold116
user
ThreeGold116
69c0f5283fd3fef6237a6935
Heng Chang
claimed_verified
2026-03-23T09:15:26.709Z
64c3698f007b906a75a14943
/avatars/da31d842225cc6074ecd904a82caf931.svg
Heng Chang
hchang95
user
hchang95
69c0f5283fd3fef6237a6936
Nuo Chen
claimed_verified
2026-03-25T10:51:29.078Z
64f5a9799bd32ae6c82e0bb4
https://cdn-avatars.huggingface.co/v1/production/uploads/64f5a9799bd32ae6c82e0bb4/j7qf4vMJhSLLKqJp5zXig.png
Chen
Nuo97
user
Nuo97
69c0f5283fd3fef6237a6937
Yuhan Li
69c0f5283fd3fef6237a6938
Huiming Fan
69c0f5283fd3fef6237a6939
Jia Li
69c0f5283fd3fef6237a693a
Ming Liu
69c0f5283fd3fef6237a693b
Bing Qin
LoopRPT: Reinforcement Pre-Training for Looped Language Models
Looped language models (LoopLMs) perform iterative latent computation to refine internal representations, offering a promising alternative to explicit chain-of-thought (CoT) reasoning. However, existing reinforcement learning (RL) paradigms primarily target output tokens, creating a structural mismatch with looped architectures whose reasoning unfolds implicitly. In this work, we propose LoopRPT, a reinforcement pre-training framework tailored for LoopLMs. By reframing next-token prediction as a next-token reasoning task, LoopRPT assigns reinforcement signals directly to latent steps using an EMA teacher reference and noisy latent rollouts. This formulation enables RL to directly shape intermediate representations, compressing effective reasoning into fewer iterations. We instantiate LoopRPT on the Ouro architecture across multiple model scales. Results demonstrate that LoopRPT consistently improves per-step representation quality, achieving Pareto dominance in accuracy-computation trade-offs. Notably, significant gains on hard tokens indicate that LoopRPT enhances early-stage reasoning rather than merely encouraging premature exits. Our findings highlight reinforcement pre-training as a principled paradigm for learning efficient latent reasoning in LoopLMs.
Reinforcement pre-training framework LoopRPT improves latent reasoning in looped language models by directly shaping intermediate representations through next-token reasoning tasks.
looped language models reinforcement learning next-token prediction latent steps EMA teacher reference noisy latent rollouts Pareto dominance chain-of-thought reasoning reinforcement pre-training