Astrolabe: Steering Forward-Process Reinforcement Learning for Distilled Autoregressive Video Models
Distilled autoregressive (AR) video models enable efficient streaming generation but frequently misalign with human visual preferences. Existing reinforcement learning (RL) frameworks are not naturally suited to these architectures, typically requiring either expensive re-distillation or solver-coupled reverse-process optimization that introduces considerable memory and computational overhead. We present Astrolabe, an efficient online RL framework tailored for distilled AR models. To overcome existing bottlenecks, we introduce a forward-process RL formulation based on negative-aware fine-tuning. By contrasting positive and negative samples directly at inference endpoints, this approach establishes an implicit policy improvement direction without requiring reverse-process unrolling. To scale this alignment to long videos, we propose a streaming training scheme that generates sequences progressively via a rolling KV-cache, applying RL updates exclusively to local clip windows while conditioning on prior context to ensure long-range coherence. Finally, to mitigate reward hacking, we integrate a multi-reward objective stabilized by uncertainty-aware selective regularization and dynamic reference updates. Extensive experiments demonstrate that our method consistently enhances generation quality across multiple distilled AR video models, serving as a robust and scalable alignment solution.
69bb739bba2b62a5fb0b0c8d
Songchun Zhang
claimed_verified
2026-03-24T13:20:55.264Z
6411c801e872ae3fb1e2c96e
/avatars/f8898dc13d700e545eedbbfab1c18353.svg
Franklin
Franklinzhang
user
Franklinzhang
69bb739bba2b62a5fb0b0c8e
Zeyue Xue
69bb739bba2b62a5fb0b0c8f
Siming Fu
69bb739bba2b62a5fb0b0c90
Jie Huang
69bb739bba2b62a5fb0b0c91
Xianghao Kong
claimed_verified
2026-03-23T10:56:34.670Z
6428fd124fe87caede856311
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/5OrNPwZkxu3Dm1IInCxML.jpeg
Xianghao Kong
refkxh
user
refkxh
69bb739bba2b62a5fb0b0c92
Y Ma
69bb739bba2b62a5fb0b0c93
Haoyang Huang
69bb739bba2b62a5fb0b0c94
Nan Duan
69bb739bba2b62a5fb0b0c95
Anyi Rao
claimed_verified
2026-03-24T09:23:08.206Z
63f8130749569335b679af62
https://cdn-avatars.huggingface.co/v1/production/uploads/63f8130749569335b679af62/vgTu23-y0UKocwAGqNMwT.jpeg
Anyi Rao
anyirao
user
anyirao
Astrolabe: Steering Forward-Process Reinforcement Learning for Distilled Autoregressive Video Models
Distilled autoregressive (AR) video models enable efficient streaming generation but frequently misalign with human visual preferences. Existing reinforcement learning (RL) frameworks are not naturally suited to these architectures, typically requiring either expensive re-distillation or solver-coupled reverse-process optimization that introduces considerable memory and computational overhead. We present Astrolabe, an efficient online RL framework tailored for distilled AR models. To overcome existing bottlenecks, we introduce a forward-process RL formulation based on negative-aware fine-tuning. By contrasting positive and negative samples directly at inference endpoints, this approach establishes an implicit policy improvement direction without requiring reverse-process unrolling. To scale this alignment to long videos, we propose a streaming training scheme that generates sequences progressively via a rolling KV-cache, applying RL updates exclusively to local clip windows while conditioning on prior context to ensure long-range coherence. Finally, to mitigate reward hacking, we integrate a multi-reward objective stabilized by uncertainty-aware selective regularization and dynamic reference updates. Extensive experiments demonstrate that our method consistently enhances generation quality across multiple distilled AR video models, serving as a robust and scalable alignment solution.
Astrolabe is an efficient online reinforcement learning framework for distilled autoregressive video models that improves generation quality through forward-process RL formulation and streaming training with multi-reward objectives.
autoregressive video models reinforcement learning distillation negative-aware fine-tuning forward-process RL streaming training KV-cache reward hacking multi-reward objective uncertainty-aware selective regularization dynamic reference updates