TempViz: On the Evaluation of Temporal Knowledge in Text-to-Image Models

TL;DR

This work addresses the paucity of temporal knowledge evaluation in text-to-image models by introducing TempViz, a dataset of 7,940 prompts across five domains designed to probe visual temporal grounding. The study simultaneously assesses five prominent T2I models through human judgments and a suite of automated evaluators, revealing that temporal cues are poorly applied and that automatic methods fail to reliably detect correct temporal depictions. TempViz's combination of prompts, reference images, and expected values enables targeted analysis of temporal reasoning in multimodal generation and provides a benchmark for future improvements in temporal grounding. The findings highlight a pressing need for better data, models, and evaluation techniques to achieve temporally coherent image synthesis in real-world prompts.

Abstract

Time alters the visual appearance of entities in our world, like objects, places, and animals. Thus, for accurately generating contextually-relevant images, knowledge and reasoning about time can be crucial (e.g., for generating a landscape in spring vs. in winter). Yet, although substantial work exists on understanding and improving temporal knowledge in natural language processing, research on how temporal phenomena appear and are handled in text-to-image (T2I) models remains scarce. We address this gap with TempViz, the first data set to holistically evaluate temporal knowledge in image generation, consisting of 7.9k prompts and more than 600 reference images. Using TempViz, we study the capabilities of five T2I models across five temporal knowledge categories. Human evaluation shows that temporal competence is generally weak, with no model exceeding 75% accuracy across categories. Towards larger-scale studies, we also examine automated evaluation methods, comparing several established approaches against human judgments. However, none of these approaches provides a reliable assessment of temporal cues - further indicating the pressing need for future research on temporal knowledge in T2I.

Figures (9)

• Figure 1: Distribution of the temporal categories within TempViz with example images of different T2I models.
• Figure 2: Accuracy per Category in % of images that correctly represent the respective concept. We present the results for each tested T2I model separately. Grey bars indicate mean accuracy per model.
• Figure 3: Example images generated by SDXL-B. We present an example from the Animals category for the subject Dwarf Hippo, shown at different stages of its life cycle that serve as temporal cues.
• Figure 4: Decompositional VQA results. We report macro F1 scores (y-axis) across categories for varying answer correctness thresholds (y-axis). Each threshold indicates the minimum percentage of correctly answered questions required to classify an image as successfully reflecting the temporal prompt. Dashed lines indicate the random baseline for each category.
• Figure 5: GPT-5 error distribution across categories and T2I models for the best performing prompting strategy.