RELIC: Interactive Video World Model with Long-Horizon Memory
A truly interactive world model requires three key ingredients: real-time long-horizon streaming, consistent spatial memory, and precise user control. However, most existing approaches address only one of these aspects in isolation, as achieving all three simultaneously is highly challenging-for example, long-term memory mechanisms often degrade real-time performance. In this work, we present RELIC, a unified framework that tackles these three challenges altogether. Given a single image and a text description, RELIC enables memory-aware, long-duration exploration of arbitrary scenes in real time. Built upon recent autoregressive video-diffusion distillation techniques, our model represents long-horizon memory using highly compressed historical latent tokens encoded with both relative actions and absolute camera poses within the KV cache. This compact, camera-aware memory structure supports implicit 3D-consistent content retrieval and enforces long-term coherence with minimal computational overhead. In parallel, we fine-tune a bidirectional teacher video model to generate sequences beyond its original 5-second training horizon, and transform it into a causal student generator using a new memory-efficient self-forcing paradigm that enables full-context distillation over long-duration teacher as well as long student self-rollouts. Implemented as a 14B-parameter model and trained on a curated Unreal Engine-rendered dataset, RELIC achieves real-time generation at 16 FPS while demonstrating more accurate action following, more stable long-horizon streaming, and more robust spatial-memory retrieval compared with prior work. These capabilities establish RELIC as a strong foundation for the next generation of interactive world modeling.
6930fd7b2d1e5b0a7d84d9d5
Yicong Hong
admin_assigned
2025-12-04T11:10:34.480Z
654ca66fe1671abcbc50a6e4
https://cdn-avatars.huggingface.co/v1/production/uploads/654ca66fe1671abcbc50a6e4/VqcnC17YcqC9Hk65_LxtE.jpeg
Yicong Hong
YicongHong
user
YicongHong
6930fd7b2d1e5b0a7d84d9d6
Yiqun Mei
6930fd7b2d1e5b0a7d84d9d7
Chongjian Ge
6930fd7b2d1e5b0a7d84d9d8
Yiran Xu
6930fd7b2d1e5b0a7d84d9d9
Yang Zhou
6930fd7b2d1e5b0a7d84d9da
Sai Bi
admin_assigned
2025-12-04T11:11:08.327Z
66303560597dc7c535168f07
/avatars/a69d52a4a198c6c8451c1d137031bf07.svg
Sai Bi
saibi
user
saibi
6930fd7b2d1e5b0a7d84d9db
Yannick Hold-Geoffroy
6930fd7b2d1e5b0a7d84d9dc
Mike Roberts
6930fd7b2d1e5b0a7d84d9dd
Matthew Fisher
6930fd7b2d1e5b0a7d84d9de
Eli Shechtman
6930fd7b2d1e5b0a7d84d9df
Kalyan Sunkavalli
admin_assigned
2025-12-04T11:11:30.379Z
67f54d1b504263bce1431e00
/avatars/ce2928e59480dbf33aacf941056dcc73.svg
Kalyan Sunkavalli
kalyanks
user
kalyanks
6930fd7b2d1e5b0a7d84d9e0
Feng Liu
6930fd7b2d1e5b0a7d84d9e1
Zhengqi Li
6930fd7b2d1e5b0a7d84d9e2
Hao Tan
RELIC: Interactive Video World Model with Long-Horizon Memory
A truly interactive world model requires three key ingredients: real-time long-horizon streaming, consistent spatial memory, and precise user control. However, most existing approaches address only one of these aspects in isolation, as achieving all three simultaneously is highly challenging-for example, long-term memory mechanisms often degrade real-time performance. In this work, we present RELIC, a unified framework that tackles these three challenges altogether. Given a single image and a text description, RELIC enables memory-aware, long-duration exploration of arbitrary scenes in real time. Built upon recent autoregressive video-diffusion distillation techniques, our model represents long-horizon memory using highly compressed historical latent tokens encoded with both relative actions and absolute camera poses within the KV cache. This compact, camera-aware memory structure supports implicit 3D-consistent content retrieval and enforces long-term coherence with minimal computational overhead. In parallel, we fine-tune a bidirectional teacher video model to generate sequences beyond its original 5-second training horizon, and transform it into a causal student generator using a new memory-efficient self-forcing paradigm that enables full-context distillation over long-duration teacher as well as long student self-rollouts. Implemented as a 14B-parameter model and trained on a curated Unreal Engine-rendered dataset, RELIC achieves real-time generation at 16 FPS while demonstrating more accurate action following, more stable long-horizon streaming, and more robust spatial-memory retrieval compared with prior work. These capabilities establish RELIC as a strong foundation for the next generation of interactive world modeling.
RELIC is a unified framework that enables real-time, memory-aware exploration of scenes by integrating long-horizon memory, spatial consistency, and user control using autoregressive video-diffusion distillation.
autoregressive video-diffusion distillation latent tokens KV cache relative actions absolute camera poses bidirectional teacher video model causal student generator memory-efficient self-forcing full-context distillation Unreal Engine-rendered dataset interactive world modeling