NaviTrace: Evaluating Embodied Navigation of Vision-Language Models
Vision-language models demonstrate unprecedented performance and
generalization across a wide range of tasks and scenarios. Integrating these
foundation models into robotic navigation systems opens pathways toward
building general-purpose robots. Yet, evaluating these models' navigation
capabilities remains constrained by costly real-world trials, overly simplified
simulations, and limited benchmarks. We introduce NaviTrace, a high-quality
Visual Question Answering benchmark where a model receives an instruction and
embodiment type (human, legged robot, wheeled robot, bicycle) and must output a
2D navigation trace in image space. Across 1000 scenarios and more than 3000
expert traces, we systematically evaluate eight state-of-the-art VLMs using a
newly introduced semantic-aware trace score. This metric combines Dynamic Time
Warping distance, goal endpoint error, and embodiment-conditioned penalties
derived from per-pixel semantics and correlates with human preferences. Our
evaluation reveals consistent gap to human performance caused by poor spatial
grounding and goal localization. NaviTrace establishes a scalable and
reproducible benchmark for real-world robotic navigation. The benchmark and
leaderboard can be found at
https://leggedrobotics.github.io/navitrace_webpage/.
69088fb9812eca10f9cc5f9c
Tim Windecker
claimed_verified
2025-11-03T20:51:32.197Z
68381e8849895d52fe3363d3
/avatars/93cc72273c37bae56bd15c476faa4484.svg
Tim Windecker
TimWindecker
user
TimWindecker
69088fb9812eca10f9cc5f9d
Manthan Patel
claimed_verified
2025-11-04T20:27:03.103Z
66fa8b357aa2514b7fb320f3
/avatars/40e6aee914ce624703d26fc61aafb119.svg
Manthan Patel
manthanhp
user
manthanhp
69088fb9812eca10f9cc5f9e
Moritz Reuss
claimed_verified
2025-11-04T20:27:01.242Z
650af93422ce64f22b619549
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/RBybzGRphbiV2MXYHrDoc.png
Moritz Reuss
mbreuss
user
mbreuss
69088fb9812eca10f9cc5f9f
Richard Schwarzkopf
69088fb9812eca10f9cc5fa0
Cesar Cadena
69088fb9812eca10f9cc5fa1
Rudolf Lioutikov
69088fb9812eca10f9cc5fa2
Marco Hutter
69088fb9812eca10f9cc5fa3
Jonas Frey
NaviTrace: Evaluating Embodied Navigation of Vision-Language Models
Vision-language models demonstrate unprecedented performance and
generalization across a wide range of tasks and scenarios. Integrating these
foundation models into robotic navigation systems opens pathways toward
building general-purpose robots. Yet, evaluating these models' navigation
capabilities remains constrained by costly real-world trials, overly simplified
simulations, and limited benchmarks. We introduce NaviTrace, a high-quality
Visual Question Answering benchmark where a model receives an instruction and
embodiment type (human, legged robot, wheeled robot, bicycle) and must output a
2D navigation trace in image space. Across 1000 scenarios and more than 3000
expert traces, we systematically evaluate eight state-of-the-art VLMs using a
newly introduced semantic-aware trace score. This metric combines Dynamic Time
Warping distance, goal endpoint error, and embodiment-conditioned penalties
derived from per-pixel semantics and correlates with human preferences. Our
evaluation reveals consistent gap to human performance caused by poor spatial
grounding and goal localization. NaviTrace establishes a scalable and
reproducible benchmark for real-world robotic navigation. The benchmark and
leaderboard can be found at
https://leggedrobotics.github.io/navitrace_webpage/.
NaviTrace is a Visual Question Answering benchmark for evaluating robotic navigation capabilities using a semantic-aware trace score across various scenarios and embodiment types.
vision-language models Visual Question Answering NaviTrace semantic-aware trace score Dynamic Time Warping distance goal endpoint error embodiment-conditioned penalties per-pixel semantics robotic navigation