LFPO: Likelihood-Free Policy Optimization for Masked Diffusion Models
Reinforcement Learning with Verifiable Rewards (RLVR) has achieved remarkable success in improving autoregressive models, especially in domains requiring correctness like mathematical reasoning and code generation. However, directly applying such paradigms to Diffusion Large Language Models (dLLMs) is fundamentally hindered by the intractability of exact likelihood computation, which forces existing methods to rely on high-variance approximations. To bridge this gap, we propose Likelihood-Free Policy Optimization (LFPO), a native framework that maps the concept of vector field flow matching to the discrete token space. Specifically, LFPO formulates alignment as geometric velocity rectification, which directly optimizes denoising logits via contrastive updates. This design effectively bypasses the errors inherent in likelihood approximation, yielding the precise gradient estimation. Furthermore, LFPO enforce consistency by predicting final solutions from intermediate steps, effectively straightening the probability flow to enable high-quality generation with significantly fewer iterations. Extensive experiments demonstrate that LFPO not only outperforms state-of-the-art baselines on code and reasoning benchmarks but also accelerates inference by approximately 20% through reduced diffusion steps.
69a8e7fb8b5b27c7ea9fa59a
Chenxing Wei
69a8e7fb8b5b27c7ea9fa59b
Jiazhen Kang
69a8e7fb8b5b27c7ea9fa59c
Hong Wang
69a8e7fb8b5b27c7ea9fa59d
Jianqing Zhang
69a8e7fb8b5b27c7ea9fa59e
Hao Jiang
69a8e7fb8b5b27c7ea9fa59f
Xiaolong Xu
69a8e7fb8b5b27c7ea9fa5a0
Ningyuan Sun
69a8e7fb8b5b27c7ea9fa5a1
Ying He
69a8e7fb8b5b27c7ea9fa5a2
F. Richard Yu
69a8e7fb8b5b27c7ea9fa5a3
Yao Shu
69a8e7fb8b5b27c7ea9fa5a4
Bo Jiang
LFPO: Likelihood-Free Policy Optimization for Masked Diffusion Models
Reinforcement Learning with Verifiable Rewards (RLVR) has achieved remarkable success in improving autoregressive models, especially in domains requiring correctness like mathematical reasoning and code generation. However, directly applying such paradigms to Diffusion Large Language Models (dLLMs) is fundamentally hindered by the intractability of exact likelihood computation, which forces existing methods to rely on high-variance approximations. To bridge this gap, we propose Likelihood-Free Policy Optimization (LFPO), a native framework that maps the concept of vector field flow matching to the discrete token space. Specifically, LFPO formulates alignment as geometric velocity rectification, which directly optimizes denoising logits via contrastive updates. This design effectively bypasses the errors inherent in likelihood approximation, yielding the precise gradient estimation. Furthermore, LFPO enforce consistency by predicting final solutions from intermediate steps, effectively straightening the probability flow to enable high-quality generation with significantly fewer iterations. Extensive experiments demonstrate that LFPO not only outperforms state-of-the-art baselines on code and reasoning benchmarks but also accelerates inference by approximately 20% through reduced diffusion steps.
LFPO enables efficient training of diffusion large language models by directly optimizing denoising logits through geometric velocity rectification, achieving faster inference and better performance on code and reasoning tasks.
reinforcement learning autoregressive models mathematical reasoning code generation diffusion large language models likelihood computation policy optimization vector field flow matching denoising logits contrastive updates geometric velocity rectification probability flow inference acceleration