TrajLoom: Dense Future Trajectory Generation from Video
Predicting future motion is crucial in video understanding and controllable video generation. Dense point trajectories are a compact, expressive motion representation, but modeling their future evolution from observed video remains challenging. We propose a framework that predicts future trajectories and visibility from past trajectories and video context. Our method has three components: (1) Grid-Anchor Offset Encoding, which reduces location-dependent bias by representing each point as an offset from its pixel-center anchor; (2) TrajLoom-VAE, which learns a compact spatiotemporal latent space for dense trajectories with masked reconstruction and a spatiotemporal consistency regularizer; and (3) TrajLoom-Flow, which generates future trajectories in latent space via flow matching, with boundary cues and on-policy K-step fine-tuning for stable sampling. We also introduce TrajLoomBench, a unified benchmark spanning real and synthetic videos with a standardized setup aligned with video-generation benchmarks. Compared with state-of-the-art methods, our approach extends the prediction horizon from 24 to 81 frames while improving motion realism and stability across datasets. The predicted trajectories directly support downstream video generation and editing. Code, model checkpoints, and datasets are available at https://trajloom.github.io/.
69c33d7195b48f16a300770a
Zewei Zhang
claimed_verified
2026-03-27T11:38:16.376Z
63e03e8942591dda0b9a0dec
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/gBl3cIV9F-IAUqBISbzpV.png
Zewei Zhang
zeweizhang
user
zeweizhang
69c33d7195b48f16a300770b
Jia Jun Cheng Xian
69c33d7195b48f16a300770c
Kaiwen Liu
69c33d7195b48f16a300770d
Ming Liang
69c33d7195b48f16a300770e
Hang Chu
69c33d7195b48f16a300770f
Jun Chen
69c33d7195b48f16a3007710
Renjie Liao
TrajLoom: Dense Future Trajectory Generation from Video
Predicting future motion is crucial in video understanding and controllable video generation. Dense point trajectories are a compact, expressive motion representation, but modeling their future evolution from observed video remains challenging. We propose a framework that predicts future trajectories and visibility from past trajectories and video context. Our method has three components: (1) Grid-Anchor Offset Encoding, which reduces location-dependent bias by representing each point as an offset from its pixel-center anchor; (2) TrajLoom-VAE, which learns a compact spatiotemporal latent space for dense trajectories with masked reconstruction and a spatiotemporal consistency regularizer; and (3) TrajLoom-Flow, which generates future trajectories in latent space via flow matching, with boundary cues and on-policy K-step fine-tuning for stable sampling. We also introduce TrajLoomBench, a unified benchmark spanning real and synthetic videos with a standardized setup aligned with video-generation benchmarks. Compared with state-of-the-art methods, our approach extends the prediction horizon from 24 to 81 frames while improving motion realism and stability across datasets. The predicted trajectories directly support downstream video generation and editing. Code, model checkpoints, and datasets are available at https://trajloom.github.io/.
A framework for predicting future motion trajectories in videos using grid-anchor encoding, variational autoencoders, and flow matching to extend prediction horizons and improve motion realism.
Grid-Anchor Offset Encoding TrajLoom-VAE TrajLoom-Flow dense point trajectories spatiotemporal latent space masked reconstruction spatiotemporal consistency regularizer flow matching boundary cues on-policy K-step fine-tuning