How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing
Recent generative models have achieved remarkable progress in image editing. However, existing systems and benchmarks remain largely text-guided. In contrast, human communication is inherently multimodal, where visual instructions such as sketches efficiently convey spatial and structural intent. To address this gap, we introduce VIBE, the Visual Instruction Benchmark for Image Editing with a three-level interaction hierarchy that captures deictic grounding, morphological manipulation, and causal reasoning. Across these levels, we curate high-quality and diverse test cases that reflect progressively increasing complexity in visual instruction following. We further propose a robust LMM-as-a-judge evaluation framework with task-specific metrics to enable scalable and fine-grained assessment. Through a comprehensive evaluation of 17 representative open-source and proprietary image editing models, we find that proprietary models exhibit early-stage visual instruction-following capabilities and consistently outperform open-source models. However, performance degrades markedly with increasing task difficulty even for the strongest systems, highlighting promising directions for future research.
69816b37ce18b18628095fa3
Huanyu Zhang
69816b37ce18b18628095fa4
Xuehai Bai
claimed_verified
2026-02-27T16:46:00.948Z
66a5f512c0c2ce1db1869144
/avatars/7f2c80a7fc587ef8e4a35bfe4c3de426.svg
bxh520
bxh520
user
bxh520
69816b37ce18b18628095fa5
Chengzu Li
69816b37ce18b18628095fa6
Chen Liang
69816b37ce18b18628095fa7
Haochen Tian
claimed_verified
2026-03-19T13:05:12.752Z
6448baa3e780dbfc89058bc3
https://cdn-avatars.huggingface.co/v1/production/uploads/6448baa3e780dbfc89058bc3/vVwmu6aFPa7YTa8qEYexB.jpeg
Haochen Tian
StarBurger
user
StarBurger
69816b37ce18b18628095fa8
Haodong Li
claimed_verified
2026-02-09T08:36:25.677Z
65ddea8b2d26e59a5a33330f
/avatars/3104ddafd6dda3c05ea9a771dbf2deeb.svg
li haodong
mickyhimself
user
mickyhimself
69816b37ce18b18628095fa9
Ruichuan An
69816b37ce18b18628095faa
Yifan Zhang
69816b37ce18b18628095fab
Anna Korhonen
69816b37ce18b18628095fac
Zhang Zhang
69816b37ce18b18628095fad
Liang Wang
69816b37ce18b18628095fae
Tieniu Tan
How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing
Recent generative models have achieved remarkable progress in image editing. However, existing systems and benchmarks remain largely text-guided. In contrast, human communication is inherently multimodal, where visual instructions such as sketches efficiently convey spatial and structural intent. To address this gap, we introduce VIBE, the Visual Instruction Benchmark for Image Editing with a three-level interaction hierarchy that captures deictic grounding, morphological manipulation, and causal reasoning. Across these levels, we curate high-quality and diverse test cases that reflect progressively increasing complexity in visual instruction following. We further propose a robust LMM-as-a-judge evaluation framework with task-specific metrics to enable scalable and fine-grained assessment. Through a comprehensive evaluation of 17 representative open-source and proprietary image editing models, we find that proprietary models exhibit early-stage visual instruction-following capabilities and consistently outperform open-source models. However, performance degrades markedly with increasing task difficulty even for the strongest systems, highlighting promising directions for future research.
Visual Instruction Benchmark for Image Editing introduces a three-level interaction hierarchy for evaluating visual instruction following capabilities in generative models.
visual instruction benchmark image editing multimodal communication deictic grounding morphological manipulation causal reasoning LMM-as-a-judge evaluation visual instruction following