Lost in Time: Clock and Calendar Understanding Challenges in Multimodal LLMs
Rohit Saxena, Aryo Pradipta Gema, Pasquale Minervini
TL;DR
The paper tackles the problem of temporal understanding from visual inputs by focusing on analogue clocks and calendars. It introduces ClockQA and CalendarQA as targeted benchmarks and evaluates seven multimodal LLMs in a zero-shot setting using metrics such as $EM$, $MAE$, and classification measures. Key findings show substantial gaps in clock-reading accuracy (e.g., $EM$ around $22.58\%$ for the best model) while calendar reasoning is stronger for some models (e.g., $Acc=0.80$ for GPT-o1) but overall remains limited. The work highlights the need for improved visual parsing, numerical reasoning, and structured temporal inference in MLLMs, guiding future benchmark design and model development.
Abstract
Understanding time from visual representations is a fundamental cognitive skill, yet it remains a challenge for multimodal large language models (MLLMs). In this work, we investigate the capabilities of MLLMs in interpreting time and date through analogue clocks and yearly calendars. To facilitate this, we curated a structured dataset comprising two subsets: 1) $\textit{ClockQA}$, which comprises various types of clock styles$-$standard, black-dial, no-second-hand, Roman numeral, and arrow-hand clocks$-$paired with time related questions; and 2) $\textit{CalendarQA}$, which consists of yearly calendar images with questions ranging from commonly known dates (e.g., Christmas, New Year's Day) to computationally derived ones (e.g., the 100th or 153rd day of the year). We aim to analyse how MLLMs can perform visual recognition, numerical reasoning, and temporal inference when presented with time-related visual data. Our evaluations show that despite recent advancements, reliably understanding time remains a significant challenge for MLLMs.