Video Depth without Video Models
Video depth estimation lifts monocular video clips to 3D by inferring dense depth at every frame. Recent advances in single-image depth estimation, brought about by the rise of large foundation models and the use of synthetic training data, have fueled a renewed interest in video depth. However, naively applying a single-image depth estimator to every frame of a video disregards temporal continuity, which not only leads to flickering but may also break when camera motion causes sudden changes in depth range. An obvious and principled solution would be to build on top of video foundation models, but these come with their own limitations; including expensive training and inference, imperfect 3D consistency, and stitching routines for the fixed-length (short) outputs. We take a step back and demonstrate how to turn a single-image latent diffusion model (LDM) into a state-of-the-art video depth estimator. Our model, which we call RollingDepth, has two main ingredients: (i) a multi-frame depth estimator that is derived from a single-image LDM and maps very short video snippets (typically frame triplets) to depth snippets. (ii) a robust, optimization-based registration algorithm that optimally assembles depth snippets sampled at various different frame rates back into a consistent video. RollingDepth is able to efficiently handle long videos with hundreds of frames and delivers more accurate depth videos than both dedicated video depth estimators and high-performing single-frame models. Project page: rollingdepth.github.io.
674d64114e69219448bcf70d
Bingxin Ke
admin_assigned
2024-12-02T09:44:14.028Z
63d90391da4f72339244c2a8
/avatars/eb0e0259c391d59739c1a205c36bb539.svg
Bingxin Ke
Bingxin
user
Bingxin
674d64114e69219448bcf70e
Dominik Narnhofer
admin_assigned
2024-12-02T09:44:18.859Z
674995c2d523f2743e6c3859
/avatars/41b60f776a41d85da9ea0b1ed7d7615b.svg
Dominik Narnhofer
dnarnhofer
user
dnarnhofer
674d64114e69219448bcf70f
Shengyu Huang
admin_assigned
2024-12-02T09:44:25.115Z
65b277cd47674826d1a08fd6
/avatars/795fb6f321aafbc56535571bc4a85dee.svg
Sheng-Yu Huang
peterjohnson
user
peterjohnson
674d64114e69219448bcf710
Lei Ke
claimed_verified
2024-12-02T17:31:56.753Z
6258a6455ea3a0a9b6de3f22
/avatars/6eeed72a97fb24465e5e65583fbe50cf.svg
Lei Ke
lkeab
user
lkeab
674d64114e69219448bcf711
Torben Peters
admin_assigned
2024-12-02T09:44:39.755Z
656d886ad848a6683a0f71c3
/avatars/494f5fd5d5fa7849850cfe21f574f12c.svg
Torben Peters
PeterTor
user
PeterTor
674d64114e69219448bcf712
Katerina Fragkiadaki
674d64114e69219448bcf713
Anton Obukhov
admin_assigned
2024-12-02T09:44:55.130Z
62f93abbc4817cfc0756b6f8
https://cdn-avatars.huggingface.co/v1/production/uploads/62f93abbc4817cfc0756b6f8/2TUC739huKDji_fKwykuh.jpeg
Anton Obukhov
toshas
user
toshas
674d64114e69219448bcf714
Konrad Schindler
claimed_verified
2025-05-15T13:45:32.605Z
6750649b7edd6a98a1bbcd06
/avatars/ba27f12d0333cf2d400d4405af7efe97.svg
Konrad Schindler
konradschindler
user
konradschindler
Video Depth without Video Models
Video depth estimation lifts monocular video clips to 3D by inferring dense depth at every frame. Recent advances in single-image depth estimation, brought about by the rise of large foundation models and the use of synthetic training data, have fueled a renewed interest in video depth. However, naively applying a single-image depth estimator to every frame of a video disregards temporal continuity, which not only leads to flickering but may also break when camera motion causes sudden changes in depth range. An obvious and principled solution would be to build on top of video foundation models, but these come with their own limitations; including expensive training and inference, imperfect 3D consistency, and stitching routines for the fixed-length (short) outputs. We take a step back and demonstrate how to turn a single-image latent diffusion model (LDM) into a state-of-the-art video depth estimator. Our model, which we call RollingDepth, has two main ingredients: (i) a multi-frame depth estimator that is derived from a single-image LDM and maps very short video snippets (typically frame triplets) to depth snippets. (ii) a robust, optimization-based registration algorithm that optimally assembles depth snippets sampled at various different frame rates back into a consistent video. RollingDepth is able to efficiently handle long videos with hundreds of frames and delivers more accurate depth videos than both dedicated video depth estimators and high-performing single-frame models. Project page: rollingdepth.github.io.
RollingDepth transforms a single-image latent diffusion model into a video depth estimator by combining multi-frame depth estimation with optimization-based registration for consistent long-video depth prediction.
latent diffusion model video depth estimation single-image depth estimation temporal continuity video foundation models depth estimation optimization-based registration frame triplets depth snippets video consistency