JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments
Current audio-visual large language models (AV-LLMs) are predominantly restricted to 2D perception, relying on RGB video and monaural audio. This design choice introduces a fundamental dimensionality mismatch that precludes reliable source localization and spatial reasoning in complex 3D environments. We address this limitation by presenting JAEGER, a framework that extends AV-LLMs to 3D space, to enable joint spatial grounding and reasoning through the integration of RGB-D observations and multi-channel first-order ambisonics. A core contribution of our work is the neural intensity vector (Neural IV), a learned spatial audio representation that encodes robust directional cues to enhance direction-of-arrival estimation, even in adverse acoustic scenarios with overlapping sources. To facilitate large-scale training and systematic evaluation, we propose SpatialSceneQA, a benchmark of 61k instruction-tuning samples curated from simulated physical environments. Extensive experiments demonstrate that our approach consistently surpasses 2D-centric baselines across diverse spatial perception and reasoning tasks, underscoring the necessity of explicit 3D modelling for advancing AI in physical environments. Our source code, pre-trained model checkpoints and datasets will be released upon acceptance.
699fbcddebfce7fbcca91b64
Zhan Liu
claimed_verified
2026-02-26T10:01:54.438Z
660bb388aeb2c22d9bd3894a
https://cdn-avatars.huggingface.co/v1/production/uploads/660bb388aeb2c22d9bd3894a/zSmagaNp6yXovj8R8-beT.jpeg
Jason Liu
liuzhan22
user
liuzhan22
699fbcddebfce7fbcca91b65
Changli Tang
699fbcddebfce7fbcca91b66
Yuxin Wang
699fbcddebfce7fbcca91b67
Zhiyuan Zhu
699fbcddebfce7fbcca91b68
Youjun Chen
699fbcddebfce7fbcca91b69
Yiwen Shao
699fbcddebfce7fbcca91b6a
Tianzi Wang
699fbcddebfce7fbcca91b6b
Lei Ke
699fbcddebfce7fbcca91b6c
Zengrui Jin
699fbcddebfce7fbcca91b6d
Chao Zhang
JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments
Current audio-visual large language models (AV-LLMs) are predominantly restricted to 2D perception, relying on RGB video and monaural audio. This design choice introduces a fundamental dimensionality mismatch that precludes reliable source localization and spatial reasoning in complex 3D environments. We address this limitation by presenting JAEGER, a framework that extends AV-LLMs to 3D space, to enable joint spatial grounding and reasoning through the integration of RGB-D observations and multi-channel first-order ambisonics. A core contribution of our work is the neural intensity vector (Neural IV), a learned spatial audio representation that encodes robust directional cues to enhance direction-of-arrival estimation, even in adverse acoustic scenarios with overlapping sources. To facilitate large-scale training and systematic evaluation, we propose SpatialSceneQA, a benchmark of 61k instruction-tuning samples curated from simulated physical environments. Extensive experiments demonstrate that our approach consistently surpasses 2D-centric baselines across diverse spatial perception and reasoning tasks, underscoring the necessity of explicit 3D modelling for advancing AI in physical environments. Our source code, pre-trained model checkpoints and datasets will be released upon acceptance.
JAEGER extends audio-visual large language models to 3D space by integrating RGB-D observations and multi-channel audio to improve spatial reasoning and source localization.
audio-visual large language models 3D space RGB-D observations multi-channel first-order ambisonics neural intensity vector direction-of-arrival estimation spatial grounding spatial reasoning SpatialSceneQA instruction-tuning