MotionStream: Real-Time Video Generation with Interactive Motion
  Controls
Current motion-conditioned video generation methods suffer from prohibitive
latency (minutes per video) and non-causal processing that prevents real-time
interaction. We present MotionStream, enabling sub-second latency with up to 29
FPS streaming generation on a single GPU. Our approach begins by augmenting a
text-to-video model with motion control, which generates high-quality videos
that adhere to the global text prompt and local motion guidance, but does not
perform inference on the fly. As such, we distill this bidirectional teacher
into a causal student through Self Forcing with Distribution Matching
Distillation, enabling real-time streaming inference. Several key challenges
arise when generating videos of long, potentially infinite time-horizons: (1)
bridging the domain gap from training on finite length and extrapolating to
infinite horizons, (2) sustaining high quality by preventing error
accumulation, and (3) maintaining fast inference, without incurring growth in
computational cost due to increasing context windows. A key to our approach is
introducing carefully designed sliding-window causal attention, combined with
attention sinks. By incorporating self-rollout with attention sinks and KV
cache rolling during training, we properly simulate inference-time
extrapolations with a fixed context window, enabling constant-speed generation
of arbitrarily long videos. Our models achieve state-of-the-art results in
motion following and video quality while being two orders of magnitude faster,
uniquely enabling infinite-length streaming. With MotionStream, users can paint
trajectories, control cameras, or transfer motion, and see results unfold in
real-time, delivering a truly interactive experience.
6909a508812eca10f9cc63de
Joonghyuk Shin
claimed_verified
2025-11-04T09:24:54.969Z
631074d895c34b95407945f0
/avatars/699baf06ec818650dec5752aca87c5b4.svg
Joonghyuk Shin
alex4727
user
alex4727
6909a508812eca10f9cc63df
Zhengqi Li
6909a508812eca10f9cc63e0
Richard Zhang
6909a508812eca10f9cc63e1
Jun-Yan Zhu
6909a508812eca10f9cc63e2
Jaesik Park
6909a508812eca10f9cc63e3
Eli Schechtman
6909a508812eca10f9cc63e4
Xun Huang
MotionStream: Real-Time Video Generation with Interactive Motion
  Controls
Current motion-conditioned video generation methods suffer from prohibitive
latency (minutes per video) and non-causal processing that prevents real-time
interaction. We present MotionStream, enabling sub-second latency with up to 29
FPS streaming generation on a single GPU. Our approach begins by augmenting a
text-to-video model with motion control, which generates high-quality videos
that adhere to the global text prompt and local motion guidance, but does not
perform inference on the fly. As such, we distill this bidirectional teacher
into a causal student through Self Forcing with Distribution Matching
Distillation, enabling real-time streaming inference. Several key challenges
arise when generating videos of long, potentially infinite time-horizons: (1)
bridging the domain gap from training on finite length and extrapolating to
infinite horizons, (2) sustaining high quality by preventing error
accumulation, and (3) maintaining fast inference, without incurring growth in
computational cost due to increasing context windows. A key to our approach is
introducing carefully designed sliding-window causal attention, combined with
attention sinks. By incorporating self-rollout with attention sinks and KV
cache rolling during training, we properly simulate inference-time
extrapolations with a fixed context window, enabling constant-speed generation
of arbitrarily long videos. Our models achieve state-of-the-art results in
motion following and video quality while being two orders of magnitude faster,
uniquely enabling infinite-length streaming. With MotionStream, users can paint
trajectories, control cameras, or transfer motion, and see results unfold in
real-time, delivering a truly interactive experience.
MotionStream enables real-time video generation with sub-second latency and up to 29 FPS by distilling a text-to-video model with motion control into a causal student using Self Forcing with Distribution Matching Distillation and sliding-window causal attention with attention sinks.
motion-conditioned video generation MotionStream text-to-video model motion control Self Forcing with Distribution Matching Distillation causal student sliding-window causal attention attention sinks self-rollout KV cache rolling motion following video quality real-time interaction