AIBench: Evaluating Visual-Logical Consistency in Academic Illustration Generation
Although image generation has boosted various applications via its rapid evolution, whether the state-of-the-art models are able to produce ready-to-use academic illustrations for papers is still largely unexplored. Directly comparing or evaluating the illustration with VLM is native but requires oracle multi-modal understanding ability, which is unreliable for long and complex texts and illustrations. To address this, we propose AIBench, the first benchmark using VQA for evaluating logic correctness of the academic illustrations and VLMs for assessing aesthetics. In detail, we designed four levels of questions proposed from a logic diagram summarized from the method part of the paper, which query whether the generated illustration aligns with the paper on different scales. Our VQA-based approach raises more accurate and detailed evaluations on visual-logical consistency while relying less on the ability of the judger VLM. With our high-quality AIBench, we conduct extensive experiments and conclude that the performance gap between models on this task is significantly larger than general ones, reflecting their various complex reasoning and high-density generation ability. Further, the logic and aesthetics are hard to optimize simultaneously as in handcrafted illustrations. Additional experiments further state that test-time scaling on both abilities significantly boosts the performance on this task.
69cfab69e02fdd68b91b7c7d
Zhaohe Liao
69cfab69e02fdd68b91b7c7e
Kaixun Jiang
69cfab69e02fdd68b91b7c7f
Zhihang Liu
69cfab69e02fdd68b91b7c80
Yujie Wei
69cfab69e02fdd68b91b7c81
Junqiu Yu
69cfab69e02fdd68b91b7c82
Quanhao Li
69cfab69e02fdd68b91b7c83
Hong-Tao Yu
69cfab69e02fdd68b91b7c84
Pandeng Li
69cfab69e02fdd68b91b7c85
Yuzheng Wang
69cfab69e02fdd68b91b7c86
Zhen Xing
69cfab69e02fdd68b91b7c87
Shiwei Zhang
69cfab69e02fdd68b91b7c88
Chen-Wei Xie
69cfab69e02fdd68b91b7c89
Yun Zheng
69cfab69e02fdd68b91b7c8a
Xihui Liu
AIBench: Evaluating Visual-Logical Consistency in Academic Illustration Generation
Although image generation has boosted various applications via its rapid evolution, whether the state-of-the-art models are able to produce ready-to-use academic illustrations for papers is still largely unexplored. Directly comparing or evaluating the illustration with VLM is native but requires oracle multi-modal understanding ability, which is unreliable for long and complex texts and illustrations. To address this, we propose AIBench, the first benchmark using VQA for evaluating logic correctness of the academic illustrations and VLMs for assessing aesthetics. In detail, we designed four levels of questions proposed from a logic diagram summarized from the method part of the paper, which query whether the generated illustration aligns with the paper on different scales. Our VQA-based approach raises more accurate and detailed evaluations on visual-logical consistency while relying less on the ability of the judger VLM. With our high-quality AIBench, we conduct extensive experiments and conclude that the performance gap between models on this task is significantly larger than general ones, reflecting their various complex reasoning and high-density generation ability. Further, the logic and aesthetics are hard to optimize simultaneously as in handcrafted illustrations. Additional experiments further state that test-time scaling on both abilities significantly boosts the performance on this task.
AIBench evaluates academic illustration quality through logic correctness and aesthetics using VQA and VLM assessments, revealing significant performance gaps and the challenge of optimizing both aspects simultaneously.
VQA VLM AIBench visual-logical consistency logic correctness aesthetics test-time scaling