From Perception to Action: An Interactive Benchmark for Vision Reasoning
Understanding the physical structure is essential for real-world applications such as embodied agents, interactive design, and long-horizon manipulation. Yet, prevailing Vision-Language Model (VLM) evaluations still center on structure-agnostic, single-turn setups (e.g., VQA), which fail to assess agents' ability to reason about how geometry, contact, and support relations jointly constrain what actions are possible in a dynamic environment. To address this gap, we introduce the Causal Hierarchy of Actions and Interactions (CHAIN) benchmark, an interactive 3D, physics-driven testbed designed to evaluate whether models can understand, plan, and execute structured action sequences grounded in physical constraints. CHAIN shifts evaluation from passive perception to active problem solving, spanning tasks such as interlocking mechanical puzzles and 3D stacking and packing. We conduct a comprehensive study of state-of-the-art VLMs and diffusion-based models under unified interactive settings. Our results show that top-performing models still struggle to internalize physical structure and causal constraints, often failing to produce reliable long-horizon plans and cannot robustly translate perceived structure into effective actions. The project is available at https://social-ai-studio.github.io/CHAIN/.
699e69fbdfbcf0b800aecafb
Yuhao Wu
claimed_verified
2026-02-25T17:29:26.632Z
63369da91ba5d5ece24118a4
/avatars/67889e1ecadb04100a77bc8b5284c6fd.svg
wuyuhao
mozhu
user
mozhu
699e69fbdfbcf0b800aecafc
Maojia Song
699e69fbdfbcf0b800aecafd
Yihuai Lan
699e69fbdfbcf0b800aecafe
Lei Wang
699e69fbdfbcf0b800aecaff
Zhiqiang Hu
claimed_verified
2026-02-25T17:29:48.417Z
637f228152229c63921119c3
https://cdn-avatars.huggingface.co/v1/production/uploads/637f228152229c63921119c3/acwXorra1r9_7i3KlBFjS.jpeg
Zhiqiang Hu
Zhiqiang007
user
Zhiqiang007
699e69fbdfbcf0b800aecb00
Yao Xiao
699e69fbdfbcf0b800aecb01
Heng Zhou
claimed_verified
2026-02-25T17:29:45.251Z
660d17d6c9be0dcd31a30b3d
/avatars/3743fe9b695c488ebe33f0d8fd607a8a.svg
Zhou Heng
henggg
user
henggg
699e69fbdfbcf0b800aecb02
Weihua Zheng
699e69fbdfbcf0b800aecb03
Dylan Raharja
699e69fbdfbcf0b800aecb04
Soujanya Poria
699e69fbdfbcf0b800aecb05
Roy Ka-Wei Lee
From Perception to Action: An Interactive Benchmark for Vision Reasoning
Understanding the physical structure is essential for real-world applications such as embodied agents, interactive design, and long-horizon manipulation. Yet, prevailing Vision-Language Model (VLM) evaluations still center on structure-agnostic, single-turn setups (e.g., VQA), which fail to assess agents' ability to reason about how geometry, contact, and support relations jointly constrain what actions are possible in a dynamic environment. To address this gap, we introduce the Causal Hierarchy of Actions and Interactions (CHAIN) benchmark, an interactive 3D, physics-driven testbed designed to evaluate whether models can understand, plan, and execute structured action sequences grounded in physical constraints. CHAIN shifts evaluation from passive perception to active problem solving, spanning tasks such as interlocking mechanical puzzles and 3D stacking and packing. We conduct a comprehensive study of state-of-the-art VLMs and diffusion-based models under unified interactive settings. Our results show that top-performing models still struggle to internalize physical structure and causal constraints, often failing to produce reliable long-horizon plans and cannot robustly translate perceived structure into effective actions. The project is available at https://social-ai-studio.github.io/CHAIN/.
Current vision-language models lack capability to understand physical structures and causal constraints needed for complex, interactive 3D tasks, as demonstrated by the CHAIN benchmark evaluating structured action planning under physical constraints.
Vision-Language Model diffusion-based models physical constraints causal constraints interactive 3D structured action sequences long-horizon planning