Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Have Multimodal Large Language Models (MLLMs) Really Learned to Tell the Time on Analog Clocks?

Tairan Fu, Miguel González, Javier Conde, Elena Merino-Gómez, Pedro Reviriego

TL;DR

The paper investigates why multimodal large language models struggle to tell time on analog clocks, proposing that both training data biases and limited generalization under multimodal reasoning contribute to the issue. By constructing a comprehensive clock-image dataset and applying fine-tuning to GPT-4o (and evaluating GPT-4.1), the authors demonstrate improvements on synthetic clocks but provide evidence that gains may rest on pattern memorization rather than true conceptual understanding. They identify two core failure factors—directional perception of clock hands and hand-function confusion—and show that fine-tuning has limited ability to overcome these when inputs are perturbed or varied. The work underscores broader generalization and robustness challenges in multimodal models and argues for training paradigms that foster abstraction and transfer, beyond mere dataset enlargement.

Abstract

Multimodal Large Language Models which can answer complex questions on an image struggle to tell the time on analog clocks. This is probably due to the lack of images with clocks at different times in their training set. In this work we explore this issue with one of the latest MLLMs: GPT-4.1 to understand why MLLMs fail to tell the time and whether fine-tuning can solve the problem. The results show how models are making progress in reading the time on analog clocks. But have they really learned to do it, or have they only learned patterns in their training datasets? In this work we put the models to the test with different clocks to illustrate the limitations of MLLMs to abstract and generalize.

Have Multimodal Large Language Models (MLLMs) Really Learned to Tell the Time on Analog Clocks?

TL;DR

The paper investigates why multimodal large language models struggle to tell time on analog clocks, proposing that both training data biases and limited generalization under multimodal reasoning contribute to the issue. By constructing a comprehensive clock-image dataset and applying fine-tuning to GPT-4o (and evaluating GPT-4.1), the authors demonstrate improvements on synthetic clocks but provide evidence that gains may rest on pattern memorization rather than true conceptual understanding. They identify two core failure factors—directional perception of clock hands and hand-function confusion—and show that fine-tuning has limited ability to overcome these when inputs are perturbed or varied. The work underscores broader generalization and robustness challenges in multimodal models and argues for training paradigms that foster abstraction and transfer, beyond mere dataset enlargement.

Abstract

Multimodal Large Language Models which can answer complex questions on an image struggle to tell the time on analog clocks. This is probably due to the lack of images with clocks at different times in their training set. In this work we explore this issue with one of the latest MLLMs: GPT-4.1 to understand why MLLMs fail to tell the time and whether fine-tuning can solve the problem. The results show how models are making progress in reading the time on analog clocks. But have they really learned to do it, or have they only learned patterns in their training datasets? In this work we put the models to the test with different clocks to illustrate the limitations of MLLMs to abstract and generalize.
Paper Structure (5 sections, 5 figures, 2 tables)

This paper contains 5 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Have MLLMs really learned to tell the time?
  3. What factors contribute to MLLM's failure in reading analog clocks?
  4. Understanding the impact of the two factors
  5. Discussion and limitations

Figures (5)

  • Figure 1: Examples of clock images found in an internet search
  • Figure 2: Effect of LLM fine-tuning on model prediction performance
  • Figure 3: Normal (left), distorted shape (middle) and modified hands (right) clock images and the times told by the original and fine-tuned versions of GPT-4.1 in each case
  • Figure 4: Examples of hand function confusion: Modified hands (left) and distorted shape (right) clocks misread by GPT-4.1 before and after fine-tuning
  • Figure 5: Performance of GPT-4.1 across clock variants and fine-tuning impact