Coupled Variational Reinforcement Learning for Language Model General Reasoning
While reinforcement learning have achieved impressive progress in language model reasoning, they are constrained by the requirement for verifiable rewards. Recent verifier-free RL methods address this limitation by utilizing the intrinsic probabilities of LLMs generating reference answers as reward signals. However, these approaches typically sample reasoning traces conditioned only on the question. This design decouples reasoning-trace sampling from answer information, leading to inefficient exploration and incoherence between traces and final answers. In this paper, we propose \b{Coupled Variational Reinforcement Learning} (CoVRL), which bridges variational inference and reinforcement learning by coupling prior and posterior distributions through a hybrid sampling strategy. By constructing and optimizing a composite distribution that integrates these two distributions, CoVRL enables efficient exploration while preserving strong thought-answer coherence. Extensive experiments on mathematical and general reasoning benchmarks show that CoVRL improves performance by 12.4\% over the base model and achieves an additional 2.3\% improvement over strong state-of-the-art verifier-free RL baselines, providing a principled framework for enhancing the general reasoning capabilities of language models.
6944dd07fbf17e708e186109
Xueru Wen
claimed_verified
2025-12-19T08:57:29.270Z
625d1201943e346492b31c7d
https://cdn-avatars.huggingface.co/v1/production/uploads/1666359066419-625d1201943e346492b31c7d.png
wenxueru
Aunderline
user
Aunderline
6944dd07fbf17e708e18610a
Jie Lou
6944dd07fbf17e708e18610b
Yanjiang Liu
6944dd07fbf17e708e18610c
Hongyu Lin
6944dd07fbf17e708e18610d
Ben He
6944dd07fbf17e708e18610e
Xianpei Han
6944dd07fbf17e708e18610f
Le Sun
6944dd07fbf17e708e186110
Yaojie Lu
6944dd07fbf17e708e186111
Debing Zhang
Coupled Variational Reinforcement Learning for Language Model General Reasoning
While reinforcement learning have achieved impressive progress in language model reasoning, they are constrained by the requirement for verifiable rewards. Recent verifier-free RL methods address this limitation by utilizing the intrinsic probabilities of LLMs generating reference answers as reward signals. However, these approaches typically sample reasoning traces conditioned only on the question. This design decouples reasoning-trace sampling from answer information, leading to inefficient exploration and incoherence between traces and final answers. In this paper, we propose \b{Coupled Variational Reinforcement Learning} (CoVRL), which bridges variational inference and reinforcement learning by coupling prior and posterior distributions through a hybrid sampling strategy. By constructing and optimizing a composite distribution that integrates these two distributions, CoVRL enables efficient exploration while preserving strong thought-answer coherence. Extensive experiments on mathematical and general reasoning benchmarks show that CoVRL improves performance by 12.4\% over the base model and achieves an additional 2.3\% improvement over strong state-of-the-art verifier-free RL baselines, providing a principled framework for enhancing the general reasoning capabilities of language models.
CoVRL, a hybrid approach combining variational inference and reinforcement learning, enhances language model reasoning by coupling prior and posterior distributions, improving performance and coherence.
reinforcement learning verifier-free RL LLMs reasoning traces variational inference prior distributions posterior distributions composite distribution efficient exploration thought-answer coherence mathematical reasoning general reasoning benchmarks