ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Tasks
Advances in diffusion, autoregressive, and hybrid models have enabled high-quality image synthesis for tasks such as text-to-image, editing, and reference-guided composition. Yet, existing benchmarks remain limited, either focus on isolated tasks, cover only narrow domains, or provide opaque scores without explaining failure modes. We introduce ImagenWorld, a benchmark of 3.6K condition sets spanning six core tasks (generation and editing, with single or multiple references) and six topical domains (artworks, photorealistic images, information graphics, textual graphics, computer graphics, and screenshots). The benchmark is supported by 20K fine-grained human annotations and an explainable evaluation schema that tags localized object-level and segment-level errors, complementing automated VLM-based metrics. Our large-scale evaluation of 14 models yields several insights: (1) models typically struggle more in editing tasks than in generation tasks, especially in local edits. (2) models excel in artistic and photorealistic settings but struggle with symbolic and text-heavy domains such as screenshots and information graphics. (3) closed-source systems lead overall, while targeted data curation (e.g., Qwen-Image) narrows the gap in text-heavy cases. (4) modern VLM-based metrics achieve Kendall accuracies up to 0.79, approximating human ranking, but fall short of fine-grained, explainable error attribution. ImagenWorld provides both a rigorous benchmark and a diagnostic tool to advance robust image generation.
69cb311122f9e9dce3fe1f35
Samin Mahdizadeh Sani
69cb311122f9e9dce3fe1f36
Max Ku
claimed_verified
2026-03-31T09:28:37.712Z
631d760344503b7227837242
https://cdn-avatars.huggingface.co/v1/production/uploads/631d760344503b7227837242/3b6JRusFX6GKJpsN9ZdeJ.png
Max Ku
vinesmsuic
user
vinesmsuic
69cb311122f9e9dce3fe1f37
Nima Jamali
69cb311122f9e9dce3fe1f38
Matina Mahdizadeh Sani
69cb311122f9e9dce3fe1f39
Paria Khoshtab
admin_assigned
2026-03-31T09:45:55.567Z
6627b04b7cd44a7df38dcdbe
/avatars/cc38224342a6a2a7158ce56ce965d56f.svg
Paria Khoshtab
Theparia
user
Theparia
69cb311122f9e9dce3fe1f3a
Wei-Chieh Sun
69cb311122f9e9dce3fe1f3b
Parnian Fazel
admin_assigned
2026-03-31T09:46:06.225Z
64c56c9b589826641ba3623c
/avatars/83a9032ae698ab8d4d38dba83cc193c6.svg
Parnian Fazel
nianpar
user
nianpar
69cb311122f9e9dce3fe1f3c
Zhi Rui Tam
69cb311122f9e9dce3fe1f3d
Thomas Chong
69cb311122f9e9dce3fe1f3e
Edisy Kin Wai Chan
69cb311122f9e9dce3fe1f3f
Donald Wai Tong Tsang
69cb311122f9e9dce3fe1f40
Chiao-Wei Hsu
69cb311122f9e9dce3fe1f41
Ting Wai Lam
claimed_verified
2026-04-07T13:39:12.056Z
67249a31b22109dd6c444229
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/l1_YzbD1Nw1X8_5-shrvs.png
Lam Ting Wai
IvanLam
user
IvanLam
69cb311122f9e9dce3fe1f42
Ho Yin Sam Ng
69cb311122f9e9dce3fe1f43
Chiafeng Chu
69cb311122f9e9dce3fe1f44
Chak-Wing Mak
69cb311122f9e9dce3fe1f45
Keming Wu
claimed_verified
2026-03-31T09:28:33.870Z
66bf00ca5b4e241fe266059d
https://cdn-avatars.huggingface.co/v1/production/uploads/66bf00ca5b4e241fe266059d/VoWPC_C4zoeT6dS699t7L.png
Keming Wu
wukeming11
user
wukeming11
69cb311122f9e9dce3fe1f46
Hiu Tung Wong
69cb311122f9e9dce3fe1f47
Yik Chun Ho
69cb311122f9e9dce3fe1f48
Chi Ruan
69cb311122f9e9dce3fe1f49
Zhuofeng Li
69cb311122f9e9dce3fe1f4a
I-Sheng Fang
69cb311122f9e9dce3fe1f4b
Shih-Ying Yeh
claimed_verified
2026-03-31T09:28:35.946Z
630593e2fca1d8d92b81d2a1
https://cdn-avatars.huggingface.co/v1/production/uploads/630593e2fca1d8d92b81d2a1/PBNjstYFGAoyx59n3eYOf.png
Shih-Ying Yeh
KBlueLeaf
user
KBlueLeaf
69cb311122f9e9dce3fe1f4c
Ho Kei Cheng
69cb311122f9e9dce3fe1f4d
Ping Nie
69cb311122f9e9dce3fe1f4e
Wenhu Chen
ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Tasks
Advances in diffusion, autoregressive, and hybrid models have enabled high-quality image synthesis for tasks such as text-to-image, editing, and reference-guided composition. Yet, existing benchmarks remain limited, either focus on isolated tasks, cover only narrow domains, or provide opaque scores without explaining failure modes. We introduce ImagenWorld, a benchmark of 3.6K condition sets spanning six core tasks (generation and editing, with single or multiple references) and six topical domains (artworks, photorealistic images, information graphics, textual graphics, computer graphics, and screenshots). The benchmark is supported by 20K fine-grained human annotations and an explainable evaluation schema that tags localized object-level and segment-level errors, complementing automated VLM-based metrics. Our large-scale evaluation of 14 models yields several insights: (1) models typically struggle more in editing tasks than in generation tasks, especially in local edits. (2) models excel in artistic and photorealistic settings but struggle with symbolic and text-heavy domains such as screenshots and information graphics. (3) closed-source systems lead overall, while targeted data curation (e.g., Qwen-Image) narrows the gap in text-heavy cases. (4) modern VLM-based metrics achieve Kendall accuracies up to 0.79, approximating human ranking, but fall short of fine-grained, explainable error attribution. ImagenWorld provides both a rigorous benchmark and a diagnostic tool to advance robust image generation.
ImagenWorld introduces a comprehensive benchmark for image generation and editing across multiple tasks and domains, featuring human annotations and explainable evaluation to assess model performance and failure modes.
diffusion models autoregressive models hybrid models text-to-image image editing reference-guided composition VLM-based metrics Kendall accuracy