Do Vision-Language Models Measure Up? Benchmarking Visual Measurement
  Reading with MeasureBench
Reading measurement instruments is effortless for humans and requires
relatively little domain expertise, yet it remains surprisingly challenging for
current vision-language models (VLMs) as we find in preliminary evaluation. In
this work, we introduce MeasureBench, a benchmark on visual measurement reading
covering both real-world and synthesized images of various types of
measurements, along with an extensible pipeline for data synthesis. Our
pipeline procedurally generates a specified type of gauge with controllable
visual appearance, enabling scalable variation in key details such as pointers,
scales, fonts, lighting, and clutter. Evaluation on popular proprietary and
open-weight VLMs shows that even the strongest frontier VLMs struggle
measurement reading in general. A consistent failure mode is indicator
localization: models can read digits or labels but misidentify the key
positions of pointers or alignments, leading to big numeric errors despite
plausible textual reasoning. We have also conducted preliminary experiments
with reinforcement learning over synthetic data, and find encouraging results
on in-domain synthetic subset but less promising for real-world images. Our
analysis highlights a fundamental limitation of current VLMs in fine-grained
spatial grounding. We hope this resource can help future advances on visually
grounded numeracy and precise spatial perception of VLMs, bridging the gap
between recognizing numbers and measuring the world.
69080eb6812eca10f9cc5de7
Fenfen Lin
69080eb6812eca10f9cc5de8
Yesheng Liu
69080eb6812eca10f9cc5de9
Haiyu Xu
claimed_verified
2025-11-04T20:27:11.380Z
68b97ed277d4f089dfded859
/avatars/ff20a41afea3bd5fd310c3e050862843.svg
Xu Haiyu
lindaxu525
user
lindaxu525
69080eb6812eca10f9cc5dea
Chen Yue
69080eb6812eca10f9cc5deb
Zheqi He
claimed_verified
2025-11-03T20:52:48.573Z
65b21047f5d76208991e463e
https://cdn-avatars.huggingface.co/v1/production/uploads/65b21047f5d76208991e463e/6ML4lLz-vUr1HdWR3Jo-L.jpeg
Zheqi He
philokey
user
philokey
69080eb6812eca10f9cc5dec
Mingxuan Zhao
69080eb6812eca10f9cc5ded
Miguel Hu Chen
claimed_verified
2025-11-05T16:25:14.975Z
66f8b76d01bae9d4a4b8ce57
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/3fHkSeKq0YMPgkdYbrfZ7.png
Miguel Hu
miguelhuchen
user
miguelhuchen
69080eb6812eca10f9cc5dee
Jiakang Liu
69080eb6812eca10f9cc5def
JG Yao
69080eb6812eca10f9cc5df0
Xi Yang
Do Vision-Language Models Measure Up? Benchmarking Visual Measurement
  Reading with MeasureBench
Reading measurement instruments is effortless for humans and requires
relatively little domain expertise, yet it remains surprisingly challenging for
current vision-language models (VLMs) as we find in preliminary evaluation. In
this work, we introduce MeasureBench, a benchmark on visual measurement reading
covering both real-world and synthesized images of various types of
measurements, along with an extensible pipeline for data synthesis. Our
pipeline procedurally generates a specified type of gauge with controllable
visual appearance, enabling scalable variation in key details such as pointers,
scales, fonts, lighting, and clutter. Evaluation on popular proprietary and
open-weight VLMs shows that even the strongest frontier VLMs struggle
measurement reading in general. A consistent failure mode is indicator
localization: models can read digits or labels but misidentify the key
positions of pointers or alignments, leading to big numeric errors despite
plausible textual reasoning. We have also conducted preliminary experiments
with reinforcement learning over synthetic data, and find encouraging results
on in-domain synthetic subset but less promising for real-world images. Our
analysis highlights a fundamental limitation of current VLMs in fine-grained
spatial grounding. We hope this resource can help future advances on visually
grounded numeracy and precise spatial perception of VLMs, bridging the gap
between recognizing numbers and measuring the world.
MeasureBench evaluates vision-language models on reading measurements from images, revealing challenges in indicator localization and fine-grained spatial grounding.
vision-language models VLMs MeasureBench data synthesis procedural generation gauge pointers scales fonts lighting clutter reinforcement learning fine-grained spatial grounding visually grounded numeracy