MMGR: Multi-Modal Generative Reasoning
Video foundation models generate visually realistic and temporally coherent content, but their reliability as world simulators depends on whether they capture physical, logical, and spatial constraints. Existing metrics such as Frechet Video Distance (FVD) emphasize perceptual quality and overlook reasoning failures, including violations of causality, physics, and global consistency. We introduce MMGR (Multi-Modal Generative Reasoning Evaluation and Benchmark), a principled evaluation framework based on five reasoning abilities: Physical, Logical, 3D Spatial, 2D Spatial, and Temporal. MMGR evaluates generative reasoning across three domains: Abstract Reasoning (ARC-AGI, Sudoku), Embodied Navigation (real-world 3D navigation and localization), and Physical Commonsense (sports and compositional interactions). MMGR applies fine-grained metrics that require holistic correctness across both video and image generation. We benchmark leading video models (Veo-3, Sora-2, Wan-2.2) and image models (Nano-banana, Nano-banana Pro, GPT-4o-image, Qwen-image), revealing strong performance gaps across domains. Models show moderate success on Physical Commonsense tasks but perform poorly on Abstract Reasoning (below 10 percent accuracy on ARC-AGI) and struggle with long-horizon spatial planning in embodied settings. Our analysis highlights key limitations in current models, including overreliance on perceptual data, weak global state consistency, and objectives that reward visual plausibility over causal correctness. MMGR offers a unified diagnostic benchmark and a path toward reasoning-aware generative world models.
69421eb65d5b2dc105274811
Zefan Cai
69421eb65d5b2dc105274812
Haoyi Qiu
claimed_verified
2025-12-18T08:00:18.327Z
644ed57a67a3dd3d0728d889
/avatars/2df18a497c2c107a48d312e8f79c4ff8.svg
Haoyi Qiu
haoyiq114
user
haoyiq114
69421eb65d5b2dc105274813
Tianyi Ma
claimed_verified
2025-12-17T10:08:32.897Z
643ebfac1a12dcf01c6b5263
https://cdn-avatars.huggingface.co/v1/production/uploads/643ebfac1a12dcf01c6b5263/thkBlRvwgf83GULvOveM6.png
Tianyi Ma
SueMintony
user
SueMintony
69421eb65d5b2dc105274814
Haozhe Zhao
69421eb65d5b2dc105274815
Gengze Zhou
claimed_verified
2025-12-17T10:08:34.841Z
6450bcd3673b2bcfaf8681af
/avatars/f5f93d780562d0772ec5dc1728945fcf.svg
Gengze Zhou
ZGZzz
user
ZGZzz
69421eb65d5b2dc105274816
Kung-Hsiang Huang
69421eb65d5b2dc105274817
Parisa Kordjamshidi
69421eb65d5b2dc105274818
Minjia Zhang
69421eb65d5b2dc105274819
Xiao Wen
69421eb65d5b2dc10527481a
Jiuxiang Gu
69421eb65d5b2dc10527481b
Nanyun Peng
69421eb65d5b2dc10527481c
Junjie Hu
MMGR: Multi-Modal Generative Reasoning
Video foundation models generate visually realistic and temporally coherent content, but their reliability as world simulators depends on whether they capture physical, logical, and spatial constraints. Existing metrics such as Frechet Video Distance (FVD) emphasize perceptual quality and overlook reasoning failures, including violations of causality, physics, and global consistency. We introduce MMGR (Multi-Modal Generative Reasoning Evaluation and Benchmark), a principled evaluation framework based on five reasoning abilities: Physical, Logical, 3D Spatial, 2D Spatial, and Temporal. MMGR evaluates generative reasoning across three domains: Abstract Reasoning (ARC-AGI, Sudoku), Embodied Navigation (real-world 3D navigation and localization), and Physical Commonsense (sports and compositional interactions). MMGR applies fine-grained metrics that require holistic correctness across both video and image generation. We benchmark leading video models (Veo-3, Sora-2, Wan-2.2) and image models (Nano-banana, Nano-banana Pro, GPT-4o-image, Qwen-image), revealing strong performance gaps across domains. Models show moderate success on Physical Commonsense tasks but perform poorly on Abstract Reasoning (below 10 percent accuracy on ARC-AGI) and struggle with long-horizon spatial planning in embodied settings. Our analysis highlights key limitations in current models, including overreliance on perceptual data, weak global state consistency, and objectives that reward visual plausibility over causal correctness. MMGR offers a unified diagnostic benchmark and a path toward reasoning-aware generative world models.
MMGR evaluates video and image models across reasoning abilities in multiple domains, revealing performance gaps and highlighting limitations in perceptual data and causal correctness.
Frechet Video Distance (FVD) MMGR Multi-Modal Generative Reasoning Evaluation and Benchmark Physical Logical 3D Spatial 2D Spatial Temporal Abstract Reasoning ARC-AGI Sudoku Embodied Navigation Physical Commonsense Veo-3 Sora-2 Wan-2.2 Nano-banana Nano-banana Pro GPT-4o-image Qwen-image perceptual quality reasoning failures causality physics global consistency holistic correctness generative reasoning world simulators