Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench

TL;DR

This paper addresses the challenge that vision-language models struggle with fine-grained instrument reading, a task requiring precise spatial grounding and numeric reasoning. It introduces MeasureBench, a benchmark with real and synthetic instrument readings across 26 instrument types and four readout designs, supplemented by a dual 2D/3D data synthesis pipeline to generate diverse, accurately labeled data. Through evaluation of 17 contemporary VLMs, the study finds unit recognition is generally strong but numeric value extraction is the main bottleneck, with composite instruments being especially difficult; reinforcement learning on synthetic data yields notable gains in the synthetic domain and modest transfer to real-world data. The work highlights a fundamental limitation in current VLMs' spatial grounding and numeracy capabilities and provides a scalable resource to drive future improvements in visually grounded numeracy and precise spatial perception.

Abstract

Reading measurement instruments is effortless for humans and requires relatively little domain expertise, yet it remains surprisingly challenging for current vision-language models (VLMs) as we find in preliminary evaluation. In this work, we introduce MeasureBench, a benchmark on visual measurement reading covering both real-world and synthesized images of various types of measurements, along with an extensible pipeline for data synthesis. Our pipeline procedurally generates a specified type of gauge with controllable visual appearance, enabling scalable variation in key details such as pointers, scales, fonts, lighting, and clutter. Evaluation on popular proprietary and open-weight VLMs shows that even the strongest frontier VLMs struggle measurement reading in general. A consistent failure mode is indicator localization: models can read digits or labels but misidentify the key positions of pointers or alignments, leading to big numeric errors despite plausible textual reasoning. We have also conducted preliminary experiments with reinforcement learning over synthetic data, and find encouraging results on in-domain synthetic subset but less promising for real-world images. Our analysis highlights a fundamental limitation of current VLMs in fine-grained spatial grounding. We hope this resource can help future advances on visually grounded numeracy and precise spatial perception of VLMs, bridging the gap between recognizing numbers and measuring the world.

Figures (15)

• Figure 1: Sampled MeasureBench examples real-world set, these four reading designs are commomly used in various measuring instruments.
• Figure 1: Key statistics of MeasureBench.
• Figure 2: Distribution of reading designs and instrument types.
• Figure 3: An hybrid measuring instrument synthesis framework.
• Figure 4: Examples of synthetic images.