CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructions
Instruction-based multimodal image manipulation has recently made rapid progress. However, existing evaluation methods lack a systematic and human-aligned framework for assessing model performance on complex and creative editing tasks. To address this gap, we propose CREval, a fully automated question-answer (QA)-based evaluation pipeline that overcomes the incompleteness and poor interpretability of opaque Multimodal Large Language Models (MLLMs) scoring. Simultaneously, we introduce CREval-Bench, a comprehensive benchmark specifically designed for creative image manipulation under complex instructions. CREval-Bench covers three categories and nine creative dimensions, comprising over 800 editing samples and 13K evaluation queries. Leveraging this pipeline and benchmark, we systematically evaluate a diverse set of state-of-the-art open and closed-source models. The results reveal that while closed-source models generally outperform open-source ones on complex and creative tasks, all models still struggle to complete such edits effectively. In addition, user studies demonstrate strong consistency between CREval's automated metrics and human judgments. Therefore, CREval provides a reliable foundation for evaluating image editing models on complex and creative image manipulation tasks, and highlights key challenges and opportunities for future research.
69ccf5061f6782218d973d67
Chonghuinan Wang
claimed_verified
2026-04-01T12:42:49.278Z
66daffbcba8010cc014aca16
/avatars/392c6a4b5ba98551cb5e70751c37f7e5.svg
ChonghuinanWang
ChonghuinanWang
user
ChonghuinanWang
69ccf5061f6782218d973d68
Zihan Chen
claimed_verified
2026-04-02T08:17:03.462Z
674c45c637652fea34f7181f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/EEWxXLmkVTYnrw1sy6oMo.png
zihan
chan121
user
chan121
69ccf5061f6782218d973d69
Yuxiang Wei
69ccf5061f6782218d973d6a
Tianyi Jiang
69ccf5061f6782218d973d6b
Xiaohe Wu
69ccf5061f6782218d973d6c
Fan Li
69ccf5061f6782218d973d6d
Wangmeng Zuo
69ccf5061f6782218d973d6e
Hongxun Yao
CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructions
Instruction-based multimodal image manipulation has recently made rapid progress. However, existing evaluation methods lack a systematic and human-aligned framework for assessing model performance on complex and creative editing tasks. To address this gap, we propose CREval, a fully automated question-answer (QA)-based evaluation pipeline that overcomes the incompleteness and poor interpretability of opaque Multimodal Large Language Models (MLLMs) scoring. Simultaneously, we introduce CREval-Bench, a comprehensive benchmark specifically designed for creative image manipulation under complex instructions. CREval-Bench covers three categories and nine creative dimensions, comprising over 800 editing samples and 13K evaluation queries. Leveraging this pipeline and benchmark, we systematically evaluate a diverse set of state-of-the-art open and closed-source models. The results reveal that while closed-source models generally outperform open-source ones on complex and creative tasks, all models still struggle to complete such edits effectively. In addition, user studies demonstrate strong consistency between CREval's automated metrics and human judgments. Therefore, CREval provides a reliable foundation for evaluating image editing models on complex and creative image manipulation tasks, and highlights key challenges and opportunities for future research.
A fully automated question-answer based evaluation pipeline and comprehensive benchmark are introduced for assessing creative image manipulation tasks under complex instructions, demonstrating strong alignment with human judgments.
Multimodal Large Language Models image manipulation evaluation pipeline creative image manipulation benchmark automated metrics human judgments