WildDet3D: Scaling Promptable 3D Detection in the Wild
Understanding objects in 3D from a single image is a cornerstone of spatial intelligence. A key step toward this goal is monocular 3D object detection--recovering the extent, location, and orientation of objects from an input RGB image. To be practical in the open world, such a detector must generalize beyond closed-set categories, support diverse prompt modalities, and leverage geometric cues when available. Progress is hampered by two bottlenecks: existing methods are designed for a single prompt type and lack a mechanism to incorporate additional geometric cues, and current 3D datasets cover only narrow categories in controlled environments, limiting open-world transfer. In this work we address both gaps. First, we introduce WildDet3D, a unified geometry-aware architecture that natively accepts text, point, and box prompts and can incorporate auxiliary depth signals at inference time. Second, we present WildDet3D-Data, the largest open 3D detection dataset to date, constructed by generating candidate 3D boxes from existing 2D annotations and retaining only human-verified ones, yielding over 1M images across 13.5K categories in diverse real-world scenes. WildDet3D establishes a new state-of-the-art across multiple benchmarks and settings. In the open-world setting, it achieves 22.6/24.8 AP3D on our newly introduced WildDet3D-Bench with text and box prompts. On Omni3D, it reaches 34.2/36.4 AP3D with text and box prompts, respectively. In zero-shot evaluation, it achieves 40.3/48.9 ODS on Argoverse 2 and ScanNet. Notably, incorporating depth cues at inference time yields substantial additional gains (+20.7 AP on average across settings).
69dc4e4b2b6b8dc66ffeebb3
Weikai Huang
69dc4e4b2b6b8dc66ffeebb4
Jieyu Zhang
69dc4e4b2b6b8dc66ffeebb5
Sijun Li
69dc4e4b2b6b8dc66ffeebb6
Taoyang Jia
69dc4e4b2b6b8dc66ffeebb7
Jiafei Duan
69dc4e4b2b6b8dc66ffeebb8
Yunqian Cheng
69dc4e4b2b6b8dc66ffeebb9
Jaemin Cho
69dc4e4b2b6b8dc66ffeebba
Mattew Wallingford
69dc4e4b2b6b8dc66ffeebbb
Rustin Soraki
69dc4e4b2b6b8dc66ffeebbc
Chris Dongjoo Kim
69dc4e4b2b6b8dc66ffeebbd
Donovan Clay
69dc4e4b2b6b8dc66ffeebbe
Taira Anderson
69dc4e4b2b6b8dc66ffeebbf
Winson Han
69dc4e4b2b6b8dc66ffeebc0
Ali Farhadi
69dc4e4b2b6b8dc66ffeebc1
Bharath Hariharan
69dc4e4b2b6b8dc66ffeebc2
Zhongzheng Ren
69dc4e4b2b6b8dc66ffeebc3
Ranjay Krishna
WildDet3D: Scaling Promptable 3D Detection in the Wild
Understanding objects in 3D from a single image is a cornerstone of spatial intelligence. A key step toward this goal is monocular 3D object detection--recovering the extent, location, and orientation of objects from an input RGB image. To be practical in the open world, such a detector must generalize beyond closed-set categories, support diverse prompt modalities, and leverage geometric cues when available. Progress is hampered by two bottlenecks: existing methods are designed for a single prompt type and lack a mechanism to incorporate additional geometric cues, and current 3D datasets cover only narrow categories in controlled environments, limiting open-world transfer. In this work we address both gaps. First, we introduce WildDet3D, a unified geometry-aware architecture that natively accepts text, point, and box prompts and can incorporate auxiliary depth signals at inference time. Second, we present WildDet3D-Data, the largest open 3D detection dataset to date, constructed by generating candidate 3D boxes from existing 2D annotations and retaining only human-verified ones, yielding over 1M images across 13.5K categories in diverse real-world scenes. WildDet3D establishes a new state-of-the-art across multiple benchmarks and settings. In the open-world setting, it achieves 22.6/24.8 AP3D on our newly introduced WildDet3D-Bench with text and box prompts. On Omni3D, it reaches 34.2/36.4 AP3D with text and box prompts, respectively. In zero-shot evaluation, it achieves 40.3/48.9 ODS on Argoverse 2 and ScanNet. Notably, incorporating depth cues at inference time yields substantial additional gains (+20.7 AP on average across settings).
A unified 3D object detection framework with a large-scale dataset enables open-world detection with multiple prompt types and geometric cue integration.
monocular 3D object detection geometry-aware architecture text prompts point prompts box prompts depth signals open-world transfer 3D detection dataset 3D object detection spatial intelligence