Efficiently Reconstructing Dynamic Scenes One D4RT at a Time
Understanding and reconstructing the complex geometry and motion of dynamic scenes from video remains a formidable challenge in computer vision. This paper introduces D4RT, a simple yet powerful feedforward model designed to efficiently solve this task. D4RT utilizes a unified transformer architecture to jointly infer depth, spatio-temporal correspondence, and full camera parameters from a single video. Its core innovation is a novel querying mechanism that sidesteps the heavy computation of dense, per-frame decoding and the complexity of managing multiple, task-specific decoders. Our decoding interface allows the model to independently and flexibly probe the 3D position of any point in space and time. The result is a lightweight and highly scalable method that enables remarkably efficient training and inference. We demonstrate that our approach sets a new state of the art, outperforming previous methods across a wide spectrum of 4D reconstruction tasks. We refer to the project webpage for animated results: https://d4rt-paper.github.io/.
69395847dfc35938ba12a223
Chuhan Zhang
69395847dfc35938ba12a224
Guillaume Le Moing
69395847dfc35938ba12a225
Skanda Koppula
69395847dfc35938ba12a226
Ignacio Rocco
69395847dfc35938ba12a227
Liliane Momeni
69395847dfc35938ba12a228
Junyu Xie
69395847dfc35938ba12a229
Shuyang Sun
69395847dfc35938ba12a22a
Rahul Sukthankar
69395847dfc35938ba12a22b
Joëlle K Barral
69395847dfc35938ba12a22c
Raia Hadsell
69395847dfc35938ba12a22d
Zoubin Ghahramani
69395847dfc35938ba12a22e
Andrew Zisserman
69395847dfc35938ba12a22f
Junlin Zhang
69395847dfc35938ba12a230
Mehdi SM Sajjadi
Efficiently Reconstructing Dynamic Scenes One D4RT at a Time
Understanding and reconstructing the complex geometry and motion of dynamic scenes from video remains a formidable challenge in computer vision. This paper introduces D4RT, a simple yet powerful feedforward model designed to efficiently solve this task. D4RT utilizes a unified transformer architecture to jointly infer depth, spatio-temporal correspondence, and full camera parameters from a single video. Its core innovation is a novel querying mechanism that sidesteps the heavy computation of dense, per-frame decoding and the complexity of managing multiple, task-specific decoders. Our decoding interface allows the model to independently and flexibly probe the 3D position of any point in space and time. The result is a lightweight and highly scalable method that enables remarkably efficient training and inference. We demonstrate that our approach sets a new state of the art, outperforming previous methods across a wide spectrum of 4D reconstruction tasks. We refer to the project webpage for animated results: https://d4rt-paper.github.io/.
D4RT, a unified transformer-based model, efficiently reconstructs 4D scenes from videos by querying 3D positions in space-time, outperforming previous methods.
feedforward model unified transformer architecture depth inference spatio-temporal correspondence camera parameters querying mechanism dense decoding task-specific decoders 4D reconstruction animated results