ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction
Embodied cognition argues that intelligence arises from sensorimotor interaction rather than passive observation. It raises an intriguing question: do modern vision-language models (VLMs), trained largely in a disembodied manner, exhibit signs of embodied cognition? We introduce ENACT, a benchmark that casts evaluation of embodied cognition as world modeling from egocentric interaction in a visual question answering (VQA) format. Framed as a partially observable Markov decision process (POMDP) whose actions are scene graph changes, ENACT comprises two complementary sequence reordering tasks: forward world modeling (reorder shuffled observations given actions) and inverse world modeling (reorder shuffled actions given observations). While conceptually simple, solving these tasks implicitly demands capabilities central to embodied cognition-affordance recognition, action-effect reasoning, embodied awareness, and interactive, long-horizon memory from partially observable egocentric input, while avoiding low-level image synthesis that could confound the evaluation. We provide a scalable pipeline that synthesizes QA pairs from robotics simulation (BEHAVIOR) and evaluates models on 8,972 QA pairs spanning long-horizon home-scale activities. Experiments reveal a performance gap between frontier VLMs and humans that widens with interaction horizon. Models consistently perform better on the inverse task than the forward one and exhibit anthropocentric biases, including a preference for right-handed actions and degradation when camera intrinsics or viewpoints deviate from human vision. Website at https://enact-embodied-cognition.github.io/.
6929d9eb4397b1ec214f6404
Qineng Wang
claimed_verified
2025-11-30T20:10:01.486Z
640c3b28c5fa12d61a50cd92
/avatars/81556de3214c848b3c3e118f50fd2968.svg
Qineng Wang
Inevitablevalor
user
Inevitablevalor
6929d9eb4397b1ec214f6405
Wenlong Huang
6929d9eb4397b1ec214f6406
Yu Zhou
6929d9eb4397b1ec214f6407
Hang Yin
6929d9eb4397b1ec214f6408
Tianwei Bao
6929d9eb4397b1ec214f6409
Jianwen Lyu
6929d9eb4397b1ec214f640a
Weiyu Liu
6929d9eb4397b1ec214f640b
Ruohan Zhang
6929d9eb4397b1ec214f640c
Jiajun Wu
6929d9eb4397b1ec214f640d
Li Fei-Fei
6929d9eb4397b1ec214f640e
Manling Li
claimed_verified
2026-04-13T07:39:44.705Z
6746a140f2ca2162e3bcfe2b
/avatars/d9d8cfb5f112e6ed7f6152fc230135d3.svg
Manling Li
ManlingLi
user
ManlingLi
ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction
Embodied cognition argues that intelligence arises from sensorimotor interaction rather than passive observation. It raises an intriguing question: do modern vision-language models (VLMs), trained largely in a disembodied manner, exhibit signs of embodied cognition? We introduce ENACT, a benchmark that casts evaluation of embodied cognition as world modeling from egocentric interaction in a visual question answering (VQA) format. Framed as a partially observable Markov decision process (POMDP) whose actions are scene graph changes, ENACT comprises two complementary sequence reordering tasks: forward world modeling (reorder shuffled observations given actions) and inverse world modeling (reorder shuffled actions given observations). While conceptually simple, solving these tasks implicitly demands capabilities central to embodied cognition-affordance recognition, action-effect reasoning, embodied awareness, and interactive, long-horizon memory from partially observable egocentric input, while avoiding low-level image synthesis that could confound the evaluation. We provide a scalable pipeline that synthesizes QA pairs from robotics simulation (BEHAVIOR) and evaluates models on 8,972 QA pairs spanning long-horizon home-scale activities. Experiments reveal a performance gap between frontier VLMs and humans that widens with interaction horizon. Models consistently perform better on the inverse task than the forward one and exhibit anthropocentric biases, including a preference for right-handed actions and degradation when camera intrinsics or viewpoints deviate from human vision. Website at https://enact-embodied-cognition.github.io/.
ENACT is a benchmark that evaluates embodied cognition in vision-language models through world modeling from egocentric interaction in a VQA format, revealing performance gaps and anthropocentric biases.
embodied cognition vision-language models VLMs benchmark world modeling visual question answering VQA partially observable Markov decision process POMDP scene graph forward world modeling inverse world modeling affordance recognition action-effect reasoning embodied awareness interactive memory long-horizon memory robotics simulation BEHAVIOR anthropocentric biases