Video Generation Models Are Good Latent Reward Models
Reward feedback learning (ReFL) has proven effective for aligning image generation with human preferences. However, its extension to video generation faces significant challenges. Existing video reward models rely on vision-language models designed for pixel-space inputs, confining ReFL optimization to near-complete denoising steps after computationally expensive VAE decoding. This pixel-space approach incurs substantial memory overhead and increased training time, and its late-stage optimization lacks early-stage supervision, refining only visual quality rather than fundamental motion dynamics and structural coherence. In this work, we show that pre-trained video generation models are naturally suited for reward modeling in the noisy latent space, as they are explicitly designed to process noisy latent representations at arbitrary timesteps and inherently preserve temporal information through their sequential modeling capabilities. Accordingly, we propose Process Reward Feedback Learning~(PRFL), a framework that conducts preference optimization entirely in latent space, enabling efficient gradient backpropagation throughout the full denoising chain without VAE decoding. Extensive experiments demonstrate that PRFL significantly improves alignment with human preferences, while achieving substantial reductions in memory consumption and training time compared to RGB ReFL.
69294e361f3fae537858b58c
Xiaoyue Mi
claimed_verified
2025-11-28T13:48:39.358Z
66f6627e08be8ab9ab0833f2
/avatars/871a0862c22e0e4f8829144953f81d85.svg
mi
mxy123
user
mxy123
69294e361f3fae537858b58d
Wenqing Yu
69294e361f3fae537858b58e
Jiesong Lian
69294e361f3fae537858b58f
Shibo Jie
69294e361f3fae537858b590
Ruizhe Zhong
69294e361f3fae537858b591
Zijun Liu
claimed_verified
2025-11-28T13:48:41.129Z
6468c76bff18750165a64df3
https://cdn-avatars.huggingface.co/v1/production/uploads/6468c76bff18750165a64df3/dHhE62SHOSJZjyU60vgh7.jpeg
Zijun Liu
BBQGOD
user
BBQGOD
69294e361f3fae537858b592
Guozhen Zhang
claimed_verified
2025-11-28T13:48:43.019Z
6684152a443492c24cdac044
/avatars/1d5abbde12a808aa743769603e494ddb.svg
Guozhen Zhang
zgzaacm
user
zgzaacm
69294e361f3fae537858b593
Zixiang Zhou
69294e361f3fae537858b594
Zhiyong Xu
69294e361f3fae537858b595
Yuan Zhou
69294e361f3fae537858b596
Qinglin Lu
69294e361f3fae537858b597
Fan Tang
Video Generation Models Are Good Latent Reward Models
Reward feedback learning (ReFL) has proven effective for aligning image generation with human preferences. However, its extension to video generation faces significant challenges. Existing video reward models rely on vision-language models designed for pixel-space inputs, confining ReFL optimization to near-complete denoising steps after computationally expensive VAE decoding. This pixel-space approach incurs substantial memory overhead and increased training time, and its late-stage optimization lacks early-stage supervision, refining only visual quality rather than fundamental motion dynamics and structural coherence. In this work, we show that pre-trained video generation models are naturally suited for reward modeling in the noisy latent space, as they are explicitly designed to process noisy latent representations at arbitrary timesteps and inherently preserve temporal information through their sequential modeling capabilities. Accordingly, we propose Process Reward Feedback Learning~(PRFL), a framework that conducts preference optimization entirely in latent space, enabling efficient gradient backpropagation throughout the full denoising chain without VAE decoding. Extensive experiments demonstrate that PRFL significantly improves alignment with human preferences, while achieving substantial reductions in memory consumption and training time compared to RGB ReFL.
PRFL optimizes video generation preferences in latent space, improving alignment with human preferences while reducing memory consumption and training time.
reward feedback learning ReFL video generation vision-language models pixel space VAE decoding latent space noisy latent representations sequential modeling gradient backpropagation PRFL