CodePercept: Code-Grounded Visual STEM Perception for MLLMs
When MLLMs fail at Science, Technology, Engineering, and Mathematics (STEM) visual reasoning, a fundamental question arises: is it due to perceptual deficiencies or reasoning limitations? Through systematic scaling analysis that independently scales perception and reasoning components, we uncover a critical insight: scaling perception consistently outperforms scaling reasoning. This reveals perception as the true lever limiting current STEM visual reasoning. Motivated by this insight, our work focuses on systematically enhancing the perception capabilities of MLLMs by establishing code as a powerful perceptual medium--executable code provides precise semantics that naturally align with the structured nature of STEM visuals. Specifically, we construct ICC-1M, a large-scale dataset comprising 1M Image-Caption-Code triplets that materializes this code-as-perception paradigm through two complementary approaches: (1) Code-Grounded Caption Generation treats executable code as ground truth for image captions, eliminating the hallucinations inherent in existing knowledge distillation methods; (2) STEM Image-to-Code Translation prompts models to generate reconstruction code, mitigating the ambiguity of natural language for perception enhancement. To validate this paradigm, we further introduce STEM2Code-Eval, a novel benchmark that directly evaluates visual perception in STEM domains. Unlike existing work relying on problem-solving accuracy as a proxy that only measures problem-relevant understanding, our benchmark requires comprehensive visual comprehension through executable code generation for image reconstruction, providing deterministic and verifiable assessment. Code is available at https://github.com/TongkunGuan/Qwen-CodePercept.
69b237b8206bad330cd7ffd1
Tongkun Guan
admin_assigned
2026-03-12T09:34:05.937Z
650d4a36cbd0c7d550d3b41b
/avatars/0f628ccbbad894098b201a188eb3bd0b.svg
TongkunGuan
TongkunGuan
user
TongkunGuan
69b237b8206bad330cd7ffd2
Zhibo Yang
69b237b8206bad330cd7ffd3
Jianqiang Wan
69b237b8206bad330cd7ffd4
Mingkun Yang
admin_assigned
2026-03-12T09:34:25.461Z
6417fa211f1f3b0fa811edc0
/avatars/fa9e1ef1472a736c2ceebe12b77d6c89.svg
Mingkun Yang
ayumiymk
user
ayumiymk
69b237b8206bad330cd7ffd5
Zhengtao Guo
69b237b8206bad330cd7ffd6
Zijian Hu
69b237b8206bad330cd7ffd7
Ruilin Luo
69b237b8206bad330cd7ffd8
Ruize Chen
admin_assigned
2026-03-12T09:34:51.598Z
66abe59bed1ef1e03b880adb
/avatars/7252bb83c8822aaa0f38c0b3300714da.svg
Ruize Chen
ruizech
user
ruizech
69b237b8206bad330cd7ffd9
Songtao Jiang
69b237b8206bad330cd7ffda
Peng Wang
69b237b8206bad330cd7ffdb
Wei Shen
69b237b8206bad330cd7ffdc
Junyang Lin
69b237b8206bad330cd7ffdd
Xiaokang Yang
CodePercept: Code-Grounded Visual STEM Perception for MLLMs
When MLLMs fail at Science, Technology, Engineering, and Mathematics (STEM) visual reasoning, a fundamental question arises: is it due to perceptual deficiencies or reasoning limitations? Through systematic scaling analysis that independently scales perception and reasoning components, we uncover a critical insight: scaling perception consistently outperforms scaling reasoning. This reveals perception as the true lever limiting current STEM visual reasoning. Motivated by this insight, our work focuses on systematically enhancing the perception capabilities of MLLMs by establishing code as a powerful perceptual medium--executable code provides precise semantics that naturally align with the structured nature of STEM visuals. Specifically, we construct ICC-1M, a large-scale dataset comprising 1M Image-Caption-Code triplets that materializes this code-as-perception paradigm through two complementary approaches: (1) Code-Grounded Caption Generation treats executable code as ground truth for image captions, eliminating the hallucinations inherent in existing knowledge distillation methods; (2) STEM Image-to-Code Translation prompts models to generate reconstruction code, mitigating the ambiguity of natural language for perception enhancement. To validate this paradigm, we further introduce STEM2Code-Eval, a novel benchmark that directly evaluates visual perception in STEM domains. Unlike existing work relying on problem-solving accuracy as a proxy that only measures problem-relevant understanding, our benchmark requires comprehensive visual comprehension through executable code generation for image reconstruction, providing deterministic and verifiable assessment. Code is available at https://github.com/TongkunGuan/Qwen-CodePercept.
MLLMs struggle with STEM visual reasoning due to perceptual limitations rather than reasoning deficiencies, and enhancing perception through code-as-perception paradigms improves performance.
MLLMs STEM visual reasoning perception capabilities code-as-perception Image-Caption-Code triplets code-grounded caption generation STEM image-to-code translation executable code visual perception benchmark evaluation