Semantic Audio-Visual Navigation in Continuous Environments
Audio-visual navigation enables embodied agents to navigate toward sound-emitting targets by leveraging both auditory and visual cues. However, most existing approaches rely on precomputed room impulse responses (RIRs) for binaural audio rendering, restricting agents to discrete grid positions and leading to spatially discontinuous observations. To establish a more realistic setting, we introduce Semantic Audio-Visual Navigation in Continuous Environments (SAVN-CE), where agents can move freely in 3D spaces and perceive temporally and spatially coherent audio-visual streams. In this setting, targets may intermittently become silent or stop emitting sound entirely, causing agents to lose goal information. To tackle this challenge, we propose MAGNet, a multimodal transformer-based model that jointly encodes spatial and semantic goal representations and integrates historical context with self-motion cues to enable memory-augmented goal reasoning. Comprehensive experiments demonstrate that MAGNet significantly outperforms state-of-the-art methods, achieving up to a 12.1\% absolute improvement in success rate. These results also highlight its robustness to short-duration sounds and long-distance navigation scenarios. The code is available at https://github.com/yichenzeng24/SAVN-CE.
69c11ceffe5f408f1bf751de
Yichen Zeng
claimed_verified
2026-03-23T11:01:12.901Z
6767b2f15b17ac358cfb42c6
/avatars/606eb37cb3e32bd76da8b4ee636c86ef.svg
Yichen Zeng
yichenzeng
user
yichenzeng
69c11ceffe5f408f1bf751df
Hebaixu Wang
69c11ceffe5f408f1bf751e0
Meng Liu
69c11ceffe5f408f1bf751e1
Yu Zhou
69c11ceffe5f408f1bf751e2
Chen Gao
69c11ceffe5f408f1bf751e3
Kehan Chen
69c11ceffe5f408f1bf751e4
Gongping Huang
Semantic Audio-Visual Navigation in Continuous Environments
Audio-visual navigation enables embodied agents to navigate toward sound-emitting targets by leveraging both auditory and visual cues. However, most existing approaches rely on precomputed room impulse responses (RIRs) for binaural audio rendering, restricting agents to discrete grid positions and leading to spatially discontinuous observations. To establish a more realistic setting, we introduce Semantic Audio-Visual Navigation in Continuous Environments (SAVN-CE), where agents can move freely in 3D spaces and perceive temporally and spatially coherent audio-visual streams. In this setting, targets may intermittently become silent or stop emitting sound entirely, causing agents to lose goal information. To tackle this challenge, we propose MAGNet, a multimodal transformer-based model that jointly encodes spatial and semantic goal representations and integrates historical context with self-motion cues to enable memory-augmented goal reasoning. Comprehensive experiments demonstrate that MAGNet significantly outperforms state-of-the-art methods, achieving up to a 12.1\% absolute improvement in success rate. These results also highlight its robustness to short-duration sounds and long-distance navigation scenarios. The code is available at https://github.com/yichenzeng24/SAVN-CE.
MAGNet, a multimodal transformer-based model, enables embodied agents to navigate audio-visual environments by jointly encoding spatial and semantic goal representations while incorporating historical context and self-motion cues for memory-augmented goal reasoning.
multimodal transformer binaural audio rendering room impulse responses embodied agents audio-visual navigation continuous environments goal representation memory-augmented reasoning self-motion cues temporal coherence