WorldCompass: Reinforcement Learning for Long-Horizon World Models
This work presents WorldCompass, a novel Reinforcement Learning (RL) post-training framework for the long-horizon, interactive video-based world models, enabling them to explore the world more accurately and consistently based on interaction signals. To effectively "steer" the world model's exploration, we introduce three core innovations tailored to the autoregressive video generation paradigm: 1) Clip-level rollout Strategy: We generate and evaluate multiple samples at a single target clip, which significantly boosts rollout efficiency and provides fine-grained reward signals. 2) Complementary Reward Functions: We design reward functions for both interaction-following accuracy and visual quality, which provide direct supervision and effectively suppress reward-hacking behaviors. 3) Efficient RL Algorithm: We employ the negative-aware fine-tuning strategy coupled with various efficiency optimizations to efficiently and effectively enhance model capacity. Evaluations on the SoTA open-source world model, WorldPlay, demonstrate that WorldCompass significantly improves interaction accuracy and visual fidelity across various scenarios.
698ab6ec1b2dc6b37d61b023
Zehan Wang
698ab6ec1b2dc6b37d61b024
Tengfei Wang
698ab6ec1b2dc6b37d61b025
Haiyu Zhang
698ab6ec1b2dc6b37d61b026
Xuhui Zuo
698ab6ec1b2dc6b37d61b027
Junta Wu
698ab6ec1b2dc6b37d61b028
Haoyuan Wang
698ab6ec1b2dc6b37d61b029
Wenqiang Sun
698ab6ec1b2dc6b37d61b02a
Zhenwei Wang
698ab6ec1b2dc6b37d61b02b
Chenjie Cao
698ab6ec1b2dc6b37d61b02c
Hengshuang Zhao
698ab6ec1b2dc6b37d61b02d
Chunchao Guo
698ab6ec1b2dc6b37d61b02e
Zhou Zhao
WorldCompass: Reinforcement Learning for Long-Horizon World Models
This work presents WorldCompass, a novel Reinforcement Learning (RL) post-training framework for the long-horizon, interactive video-based world models, enabling them to explore the world more accurately and consistently based on interaction signals. To effectively "steer" the world model's exploration, we introduce three core innovations tailored to the autoregressive video generation paradigm: 1) Clip-level rollout Strategy: We generate and evaluate multiple samples at a single target clip, which significantly boosts rollout efficiency and provides fine-grained reward signals. 2) Complementary Reward Functions: We design reward functions for both interaction-following accuracy and visual quality, which provide direct supervision and effectively suppress reward-hacking behaviors. 3) Efficient RL Algorithm: We employ the negative-aware fine-tuning strategy coupled with various efficiency optimizations to efficiently and effectively enhance model capacity. Evaluations on the SoTA open-source world model, WorldPlay, demonstrate that WorldCompass significantly improves interaction accuracy and visual fidelity across various scenarios.
WorldCompass enhances long-horizon video-based world models through reinforcement learning post-training with clip-level rollouts, complementary rewards, and efficient RL algorithms.
Reinforcement Learning world models video generation rollout strategy reward functions reward-hacking negative-aware fine-tuning efficiency optimizations