VisJudge-Bench: Aesthetics and Quality Assessment of Visualizations
Visualization, a domain-specific yet widely used form of imagery, is an
effective way to turn complex datasets into intuitive insights, and its value
depends on whether data are faithfully represented, clearly communicated, and
aesthetically designed. However, evaluating visualization quality is
challenging: unlike natural images, it requires simultaneous judgment across
data encoding accuracy, information expressiveness, and visual aesthetics.
Although multimodal large language models (MLLMs) have shown promising
performance in aesthetic assessment of natural images, no systematic benchmark
exists for measuring their capabilities in evaluating visualizations. To
address this, we propose VisJudge-Bench, the first comprehensive benchmark for
evaluating MLLMs' performance in assessing visualization aesthetics and
quality. It contains 3,090 expert-annotated samples from real-world scenarios,
covering single visualizations, multiple visualizations, and dashboards across
32 chart types. Systematic testing on this benchmark reveals that even the most
advanced MLLMs (such as GPT-5) still exhibit significant gaps compared to human
experts in judgment, with a Mean Absolute Error (MAE) of 0.551 and a
correlation with human ratings of only 0.429. To address this issue, we propose
VisJudge, a model specifically designed for visualization aesthetics and
quality assessment. Experimental results demonstrate that VisJudge
significantly narrows the gap with human judgment, reducing the MAE to 0.442 (a
19.8% reduction) and increasing the consistency with human experts to 0.681 (a
58.7% improvement) compared to GPT-5. The benchmark is available at
https://github.com/HKUSTDial/VisJudgeBench.
6901cb91646208eac0d1f5a8
Yupeng Xie
6901cb91646208eac0d1f5a9
Zhiyang Zhang
6901cb91646208eac0d1f5aa
Yifan Wu
6901cb91646208eac0d1f5ab
Sirong Lu
6901cb91646208eac0d1f5ac
Jiayi Zhang
claimed_verified
2025-11-21T09:02:46.308Z
65f40e83653c231cbaf7defe
/avatars/afa5ce72324112739e539865c9aee26b.svg
Jiayi Zhang
didiforhugface
user
didiforhugface
6901cb91646208eac0d1f5ad
Zhaoyang Yu
claimed_verified
2025-11-20T08:50:31.723Z
640dc84b474aa6f89554d518
/avatars/9fcee1023ed5c6cddb3c342e19f18295.svg
Zhaoyang Yu
MoshiQAQ
user
MoshiQAQ
6901cb91646208eac0d1f5ae
Jinlin Wang
6901cb91646208eac0d1f5af
Sirui Hong
6901cb91646208eac0d1f5b0
Bang Liu
6901cb91646208eac0d1f5b1
Chenglin Wu
6901cb91646208eac0d1f5b2
Yuyu Luo
VisJudge-Bench: Aesthetics and Quality Assessment of Visualizations
Visualization, a domain-specific yet widely used form of imagery, is an
effective way to turn complex datasets into intuitive insights, and its value
depends on whether data are faithfully represented, clearly communicated, and
aesthetically designed. However, evaluating visualization quality is
challenging: unlike natural images, it requires simultaneous judgment across
data encoding accuracy, information expressiveness, and visual aesthetics.
Although multimodal large language models (MLLMs) have shown promising
performance in aesthetic assessment of natural images, no systematic benchmark
exists for measuring their capabilities in evaluating visualizations. To
address this, we propose VisJudge-Bench, the first comprehensive benchmark for
evaluating MLLMs' performance in assessing visualization aesthetics and
quality. It contains 3,090 expert-annotated samples from real-world scenarios,
covering single visualizations, multiple visualizations, and dashboards across
32 chart types. Systematic testing on this benchmark reveals that even the most
advanced MLLMs (such as GPT-5) still exhibit significant gaps compared to human
experts in judgment, with a Mean Absolute Error (MAE) of 0.551 and a
correlation with human ratings of only 0.429. To address this issue, we propose
VisJudge, a model specifically designed for visualization aesthetics and
quality assessment. Experimental results demonstrate that VisJudge
significantly narrows the gap with human judgment, reducing the MAE to 0.442 (a
19.8% reduction) and increasing the consistency with human experts to 0.681 (a
58.7% improvement) compared to GPT-5. The benchmark is available at
https://github.com/HKUSTDial/VisJudgeBench.
VisJudge-Bench is a benchmark for evaluating MLLMs' performance in assessing visualization aesthetics and quality, revealing gaps compared to human experts and demonstrating improvements with the VisJudge model.
multimodal large language models MLLMs VisJudge-Bench expert-annotated samples chart types Mean Absolute Error MAE correlation with human ratings VisJudge