PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language
  Models in Physical Environments
Visual reasoning in multimodal large language models (MLLMs) has primarily
been studied in static, fully observable settings, limiting their effectiveness
in real-world environments where information is often incomplete due to
occlusion or limited field of view. Humans, in contrast, actively explore and
interact with their environment-moving, examining, and manipulating objects-to
gather information through a closed-loop process integrating perception,
reasoning, and action. Inspired by this human capability, we introduce the
Active Visual Reasoning (AVR) task, extending visual reasoning to partially
observable, interactive environments. AVR necessitates agents to: (1) actively
acquire information via sequential physical actions, (2) integrate observations
across multiple steps for coherent reasoning, and (3) dynamically adjust
decisions based on evolving visual feedback. To rigorously evaluate AVR, we
introduce CLEVR-AVR, a simulation benchmark featuring multi-round interactive
environments designed to assess both reasoning correctness and
information-gathering efficiency. We present AVR-152k, a large-scale dataset
that offers rich Chain-of-Thought (CoT) annotations detailing iterative
reasoning for uncertainty identification, action-conditioned information gain
prediction, and information-maximizing action selection, crucial for training
agents in a higher-order Markov Decision Process. Building on this, we develop
PhysVLM-AVR, an MLLM achieving state-of-the-art performance on CLEVR-AVR,
embodied reasoning (OpenEQA, RoboVQA), and passive visual reasoning (GeoMath,
Geometry30K). Our analysis also reveals that current embodied MLLMs, despite
detecting information incompleteness, struggle to actively acquire and
integrate new information through interaction, highlighting a fundamental gap
in active reasoning capabilities.
68ff1bad6cdff8b857f4726a
Weijie Zhou
claimed_verified
2025-10-27T10:24:53.173Z
64c8fa690d3d1b209d2821a1
/avatars/0c0e6d9590ea7b487c6e909784d3d99a.svg
weijie zhou
JettZhou
user
JettZhou
68ff1bad6cdff8b857f4726b
Xuantang Xiong
68ff1bad6cdff8b857f4726c
Yi Peng
68ff1bad6cdff8b857f4726d
Manli Tao
68ff1bad6cdff8b857f4726e
Chaoyang Zhao
68ff1bad6cdff8b857f4726f
Honghui Dong
68ff1bad6cdff8b857f47270
Ming Tang
68ff1bad6cdff8b857f47271
Jinqiao Wang
PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language
  Models in Physical Environments
Visual reasoning in multimodal large language models (MLLMs) has primarily
been studied in static, fully observable settings, limiting their effectiveness
in real-world environments where information is often incomplete due to
occlusion or limited field of view. Humans, in contrast, actively explore and
interact with their environment-moving, examining, and manipulating objects-to
gather information through a closed-loop process integrating perception,
reasoning, and action. Inspired by this human capability, we introduce the
Active Visual Reasoning (AVR) task, extending visual reasoning to partially
observable, interactive environments. AVR necessitates agents to: (1) actively
acquire information via sequential physical actions, (2) integrate observations
across multiple steps for coherent reasoning, and (3) dynamically adjust
decisions based on evolving visual feedback. To rigorously evaluate AVR, we
introduce CLEVR-AVR, a simulation benchmark featuring multi-round interactive
environments designed to assess both reasoning correctness and
information-gathering efficiency. We present AVR-152k, a large-scale dataset
that offers rich Chain-of-Thought (CoT) annotations detailing iterative
reasoning for uncertainty identification, action-conditioned information gain
prediction, and information-maximizing action selection, crucial for training
agents in a higher-order Markov Decision Process. Building on this, we develop
PhysVLM-AVR, an MLLM achieving state-of-the-art performance on CLEVR-AVR,
embodied reasoning (OpenEQA, RoboVQA), and passive visual reasoning (GeoMath,
Geometry30K). Our analysis also reveals that current embodied MLLMs, despite
detecting information incompleteness, struggle to actively acquire and
integrate new information through interaction, highlighting a fundamental gap
in active reasoning capabilities.
Active Visual Reasoning (AVR) extends visual reasoning to interactive, partially observable environments, requiring agents to sequentially gather and integrate information for coherent decision-making, as evaluated by the CLEVR-AVR benchmark.
Active Visual Reasoning AVR CLEVR-AVR Chain-of-Thought CoT embodied reasoning OpenEQA RoboVQA passive visual reasoning GeoMath Geometry30K Markov Decision Process