Track4World: Feedforward World-centric Dense 3D Tracking of All Pixels
Estimating the 3D trajectory of every pixel from a monocular video is crucial and promising for a comprehensive understanding of the 3D dynamics of videos. Recent monocular 3D tracking works demonstrate impressive performance, but are limited to either tracking sparse points on the first frame or a slow optimization-based framework for dense tracking. In this paper, we propose a feedforward model, called Track4World, enabling an efficient holistic 3D tracking of every pixel in the world-centric coordinate system. Built on the global 3D scene representation encoded by a VGGT-style ViT, Track4World applies a novel 3D correlation scheme to simultaneously estimate the pixel-wise 2D and 3D dense flow between arbitrary frame pairs. The estimated scene flow, along with the reconstructed 3D geometry, enables subsequent efficient 3D tracking of every pixel of this video. Extensive experiments on multiple benchmarks demonstrate that our approach consistently outperforms existing methods in 2D/3D flow estimation and 3D tracking, highlighting its robustness and scalability for real-world 4D reconstruction tasks.
69a7ae0d2833ff1122ff64fa
Jiahao Lu
69a7ae0d2833ff1122ff64fb
Jiayi Xu
69a7ae0d2833ff1122ff64fc
Wenbo Hu
69a7ae0d2833ff1122ff64fd
Ruijie Zhu
claimed_verified
2026-03-13T08:52:17.255Z
6697ac8427e4e21a3a92da27
https://cdn-avatars.huggingface.co/v1/production/uploads/6697ac8427e4e21a3a92da27/9vn07-1_BBDk9zfDtDpcG.png
Ruijie Zhu
RuijieZhu
user
RuijieZhu
69a7ae0d2833ff1122ff64fe
Chengfeng Zhao
69a7ae0d2833ff1122ff64ff
Sai-Kit Yeung
69a7ae0d2833ff1122ff6500
Ying Shan
69a7ae0d2833ff1122ff6501
Yuan Liu
Track4World: Feedforward World-centric Dense 3D Tracking of All Pixels
Estimating the 3D trajectory of every pixel from a monocular video is crucial and promising for a comprehensive understanding of the 3D dynamics of videos. Recent monocular 3D tracking works demonstrate impressive performance, but are limited to either tracking sparse points on the first frame or a slow optimization-based framework for dense tracking. In this paper, we propose a feedforward model, called Track4World, enabling an efficient holistic 3D tracking of every pixel in the world-centric coordinate system. Built on the global 3D scene representation encoded by a VGGT-style ViT, Track4World applies a novel 3D correlation scheme to simultaneously estimate the pixel-wise 2D and 3D dense flow between arbitrary frame pairs. The estimated scene flow, along with the reconstructed 3D geometry, enables subsequent efficient 3D tracking of every pixel of this video. Extensive experiments on multiple benchmarks demonstrate that our approach consistently outperforms existing methods in 2D/3D flow estimation and 3D tracking, highlighting its robustness and scalability for real-world 4D reconstruction tasks.
A feedforward model called Track4World enables efficient holistic 3D tracking of every pixel in a video by utilizing a global 3D scene representation and novel 3D correlation scheme for dense flow estimation.
VGGT-style ViT 3D correlation scheme pixel-wise 2D and 3D dense flow scene flow 3D geometry reconstruction 4D reconstruction