Multi-View 3D Point Tracking
We introduce the first data-driven multi-view 3D point tracker, designed to
track arbitrary points in dynamic scenes using multiple camera views. Unlike
existing monocular trackers, which struggle with depth ambiguities and
occlusion, or prior multi-camera methods that require over 20 cameras and
tedious per-sequence optimization, our feed-forward model directly predicts 3D
correspondences using a practical number of cameras (e.g., four), enabling
robust and accurate online tracking. Given known camera poses and either
sensor-based or estimated multi-view depth, our tracker fuses multi-view
features into a unified point cloud and applies k-nearest-neighbors correlation
alongside a transformer-based update to reliably estimate long-range 3D
correspondences, even under occlusion. We train on 5K synthetic multi-view
Kubric sequences and evaluate on two real-world benchmarks: Panoptic Studio and
DexYCB, achieving median trajectory errors of 3.1 cm and 2.0 cm, respectively.
Our method generalizes well to diverse camera setups of 1-8 views with varying
vantage points and video lengths of 24-150 frames. By releasing our tracker
alongside training and evaluation datasets, we aim to set a new standard for
multi-view 3D tracking research and provide a practical tool for real-world
applications. Project page available at https://ethz-vlg.github.io/mvtracker.
68b145d7b19c540001484a0c
Frano Rajič
68b145d7b19c540001484a0d
Haofei Xu
68b145d7b19c540001484a0e
Marko Mihajlovic
68b145d7b19c540001484a0f
Siyuan Li
68b145d7b19c540001484a10
Irem Demir
68b145d7b19c540001484a11
Emircan Gündoğdu
68b145d7b19c540001484a12
Lei Ke
claimed_verified
2025-12-12T09:18:44.537Z
6258a6455ea3a0a9b6de3f22
/avatars/6eeed72a97fb24465e5e65583fbe50cf.svg
Lei Ke
lkeab
user
lkeab
68b145d7b19c540001484a13
Sergey Prokudin
68b145d7b19c540001484a14
Marc Pollefeys
68b145d7b19c540001484a15
Siyu Tang
Multi-View 3D Point Tracking
We introduce the first data-driven multi-view 3D point tracker, designed to
track arbitrary points in dynamic scenes using multiple camera views. Unlike
existing monocular trackers, which struggle with depth ambiguities and
occlusion, or prior multi-camera methods that require over 20 cameras and
tedious per-sequence optimization, our feed-forward model directly predicts 3D
correspondences using a practical number of cameras (e.g., four), enabling
robust and accurate online tracking. Given known camera poses and either
sensor-based or estimated multi-view depth, our tracker fuses multi-view
features into a unified point cloud and applies k-nearest-neighbors correlation
alongside a transformer-based update to reliably estimate long-range 3D
correspondences, even under occlusion. We train on 5K synthetic multi-view
Kubric sequences and evaluate on two real-world benchmarks: Panoptic Studio and
DexYCB, achieving median trajectory errors of 3.1 cm and 2.0 cm, respectively.
Our method generalizes well to diverse camera setups of 1-8 views with varying
vantage points and video lengths of 24-150 frames. By releasing our tracker
alongside training and evaluation datasets, we aim to set a new standard for
multi-view 3D tracking research and provide a practical tool for real-world
applications. Project page available at https://ethz-vlg.github.io/mvtracker.
A multi-view 3D point tracker using a feed-forward model with transformers and k-nearest-neighbors achieves robust tracking with fewer cameras and less optimization compared to existing methods.
multi-view 3D point tracker feed-forward model transformer-based update k-nearest-neighbors multi-view features unified point cloud long-range 3D correspondences Panoptic Studio DexYCB synthetic multi-view Kubric sequences