MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answering

TL;DR

MTBench addresses the need for robust evaluation of large language models on integrated multimodal data by pairing time-series with aligned textual narratives in finance and weather. It introduces a multi-task benchmark encompassing forecasting, semantic trend analysis, technical-indicator prediction, and news-driven QA, under a dataset collection and alignment pipeline that enables controllable input lengths. Experimental results show that incorporating textual context generally enhances time-series predictions and cross-modal reasoning, though long-horizon temporal dependencies and cross-modal fusion remain challenging, with some models exhibiting output-length and horizon-specific limitations. The benchmark provides a flexible, scalable framework for domain-specific multimodal evaluation and offers a path toward more capable temporal-reasoning systems.

Abstract

Understanding the relationship between textual news and time-series evolution is a critical yet under-explored challenge in applied data science. While multimodal learning has gained traction, existing multimodal time-series datasets fall short in evaluating cross-modal reasoning and complex question answering, which are essential for capturing complex interactions between narrative information and temporal patterns. To bridge this gap, we introduce Multimodal Time Series Benchmark (MTBench), a large-scale benchmark designed to evaluate large language models (LLMs) on time series and text understanding across financial and weather domains. MTbench comprises paired time series and textual data, including financial news with corresponding stock price movements and weather reports aligned with historical temperature records. Unlike existing benchmarks that focus on isolated modalities, MTbench provides a comprehensive testbed for models to jointly reason over structured numerical trends and unstructured textual narratives. The richness of MTbench enables formulation of diverse tasks that require a deep understanding of both text and time-series data, including time-series forecasting, semantic and technical trend analysis, and news-driven question answering (QA). These tasks target the model's ability to capture temporal dependencies, extract key insights from textual context, and integrate cross-modal information. We evaluate state-of-the-art LLMs on MTbench, analyzing their effectiveness in modeling the complex relationships between news narratives and temporal patterns. Our findings reveal significant challenges in current models, including difficulties in capturing long-term dependencies, interpreting causality in financial and weather trends, and effectively fusing multimodal information.

Figures (9)

• Figure 1: An overview of tasks in MTBench.
• Figure 2: Pipeline of collecting news and time-series pairs.
• Figure 3: Geographical distribution of weather stations
• Figure 4: Article token counts distribution of financial (left) and weather (right) dataset.
• Figure 5: Distribution across article content types (left), temporal impact range (middle), and sentiment (right).