SO-Bench: A Structural Output Evaluation of Multimodal LLMs
Multimodal large language models (MLLMs) are increasingly deployed in real-world, agentic settings where outputs must not only be correct, but also conform to predefined data schemas. Despite recent progress in structured generation in textual domain, there is still no benchmark that systematically evaluates schema-grounded information extraction and reasoning over visual inputs. In this work, we conduct a comprehensive study of visual structural output capabilities for MLLMs with our carefully designed SO-Bench benchmark. Covering four visual domains, including UI screens, natural images, documents, and charts, SO-Bench is built from over 6.5K diverse JSON schemas and 1.8K curated image-schema pairs with human-verified quality. Benchmarking experiments on open-sourced and frontier proprietary models reveal persistent gaps in predicting accurate, schema compliant outputs, highlighting the need for better multimodal structured reasoning. Beyond benchmarking, we further conduct training experiments to largely improve the model's structured output capability. We plan to make the benchmark available to the community.
692d0d2b4397b1ec214f67d9
Di Feng
692d0d2b4397b1ec214f67da
Kaixin Ma
692d0d2b4397b1ec214f67db
Feng Nan
692d0d2b4397b1ec214f67dc
Haofeng Chen
692d0d2b4397b1ec214f67dd
Bohan Zhai
692d0d2b4397b1ec214f67de
David Griffiths
692d0d2b4397b1ec214f67df
Mingfei Gao
692d0d2b4397b1ec214f67e0
Zhe Gan
692d0d2b4397b1ec214f67e1
Eshan Verma
692d0d2b4397b1ec214f67e2
Yinfei Yang
692d0d2b4397b1ec214f67e3
Zhifeng Chen
692d0d2b4397b1ec214f67e4
Afshin Dehghan
SO-Bench: A Structural Output Evaluation of Multimodal LLMs
Multimodal large language models (MLLMs) are increasingly deployed in real-world, agentic settings where outputs must not only be correct, but also conform to predefined data schemas. Despite recent progress in structured generation in textual domain, there is still no benchmark that systematically evaluates schema-grounded information extraction and reasoning over visual inputs. In this work, we conduct a comprehensive study of visual structural output capabilities for MLLMs with our carefully designed SO-Bench benchmark. Covering four visual domains, including UI screens, natural images, documents, and charts, SO-Bench is built from over 6.5K diverse JSON schemas and 1.8K curated image-schema pairs with human-verified quality. Benchmarking experiments on open-sourced and frontier proprietary models reveal persistent gaps in predicting accurate, schema compliant outputs, highlighting the need for better multimodal structured reasoning. Beyond benchmarking, we further conduct training experiments to largely improve the model's structured output capability. We plan to make the benchmark available to the community.
A benchmark evaluates schema-grounded information extraction and reasoning over visual inputs for multimodal large language models, revealing gaps and guiding future improvements.
multimodal large language models schema-grounded information extraction reasoning over visual inputs SO-Bench JSON schemas image-schema pairs structured generation multimodal structured reasoning