From Masks to Pixels and Meaning: A New Taxonomy, Benchmark, and Metrics for VLM Image Tampering
Existing tampering detection benchmarks largely rely on object masks, which severely misalign with the true edit signal: many pixels inside a mask are untouched or only trivially modified, while subtle yet consequential edits outside the mask are treated as natural. We reformulate VLM image tampering from coarse region labels to a pixel-grounded, meaning and language-aware task. First, we introduce a taxonomy spanning edit primitives (replace/remove/splice/inpaint/attribute/colorization, etc.) and their semantic class of tampered object, linking low-level changes to high-level understanding. Second, we release a new benchmark with per-pixel tamper maps and paired category supervision to evaluate detection and classification within a unified protocol. Third, we propose a training framework and evaluation metrics that quantify pixel-level correctness with localization to assess confidence or prediction on true edit intensity, and further measure tamper meaning understanding via semantics-aware classification and natural language descriptions for the predicted regions. We also re-evaluate the existing strong segmentation/localization baselines on recent strong tamper detectors and reveal substantial over- and under-scoring using mask-only metrics, and expose failure modes on micro-edits and off-mask changes. Our framework advances the field from masks to pixels, meanings and language descriptions, establishing a rigorous standard for tamper localization, semantic classification and description. Code and benchmark data are available at https://github.com/VILA-Lab/PIXAR.
69c1a80cc105f45cb0481d1a
Xinyi Shang
69c1a80cc105f45cb0481d1b
Yi Tang
69c1a80cc105f45cb0481d1c
Jiacheng Cui
69c1a80cc105f45cb0481d1d
Ahmed Elhagry
69c1a80cc105f45cb0481d1e
Salwa K. Al Khatib
69c1a80cc105f45cb0481d1f
Sondos Mahmoud Bsharat
69c1a80cc105f45cb0481d20
Jiacheng Liu
69c1a80cc105f45cb0481d21
Xiaohan Zhao
69c1a80cc105f45cb0481d22
Jing-Hao Xue
69c1a80cc105f45cb0481d23
Hao Li
69c1a80cc105f45cb0481d24
Salman Khan
69c1a80cc105f45cb0481d25
Zhiqiang Shen
From Masks to Pixels and Meaning: A New Taxonomy, Benchmark, and Metrics for VLM Image Tampering
Existing tampering detection benchmarks largely rely on object masks, which severely misalign with the true edit signal: many pixels inside a mask are untouched or only trivially modified, while subtle yet consequential edits outside the mask are treated as natural. We reformulate VLM image tampering from coarse region labels to a pixel-grounded, meaning and language-aware task. First, we introduce a taxonomy spanning edit primitives (replace/remove/splice/inpaint/attribute/colorization, etc.) and their semantic class of tampered object, linking low-level changes to high-level understanding. Second, we release a new benchmark with per-pixel tamper maps and paired category supervision to evaluate detection and classification within a unified protocol. Third, we propose a training framework and evaluation metrics that quantify pixel-level correctness with localization to assess confidence or prediction on true edit intensity, and further measure tamper meaning understanding via semantics-aware classification and natural language descriptions for the predicted regions. We also re-evaluate the existing strong segmentation/localization baselines on recent strong tamper detectors and reveal substantial over- and under-scoring using mask-only metrics, and expose failure modes on micro-edits and off-mask changes. Our framework advances the field from masks to pixels, meanings and language descriptions, establishing a rigorous standard for tamper localization, semantic classification and description. Code and benchmark data are available at https://github.com/VILA-Lab/PIXAR.
Researchers reformulate vision-language model tampering detection by shifting from object masks to pixel-level analysis with semantic understanding, introducing a new benchmark and evaluation framework that assesses both localization accuracy and semantic comprehension of image modifications.
tampering detection vision-language models object masks pixel-grounded edit primitives semantic class tampered object tamper maps category supervision training framework evaluation metrics pixel-level correctness localization semantics-aware classification natural language descriptions segmentation localization baselines micro-edits off-mask changes