Time-MMD: Multi-Domain Multimodal Dataset for Time Series Analysis

TL;DR

Time-MMD introduces the first multi-domain multimodal time-series dataset, pairing numerical signals with exogenous textual information across nine domains to enable richer TSA. It enforces fine-grained modality alignment, prevents data contamination, and provides up-to-date cutoffs, complemented by MM-TSFlib, a pipeline that integrates LLM-based text modeling with traditional TSF models. Across over 1,000 experiments, multimodal forecasting consistently outperforms unimodal baselines, with average MSE improvements around 15% and up to 40% in text-rich domains. This work offers a practical path to advancing TSA by foregrounding textual context and providing a reusable open-source toolkit and dataset for researchers and practitioners.

Abstract

Time series data are ubiquitous across a wide range of real-world domains. While real-world time series analysis (TSA) requires human experts to integrate numerical series data with multimodal domain-specific knowledge, most existing TSA models rely solely on numerical data, overlooking the significance of information beyond numerical series. This oversight is due to the untapped potential of textual series data and the absence of a comprehensive, high-quality multimodal dataset. To overcome this obstacle, we introduce Time-MMD, the first multi-domain, multimodal time series dataset covering 9 primary data domains. Time-MMD ensures fine-grained modality alignment, eliminates data contamination, and provides high usability. Additionally, we develop MM-TSFlib, the first-cut multimodal time-series forecasting (TSF) library, seamlessly pipelining multimodal TSF evaluations based on Time-MMD for in-depth analyses. Extensive experiments conducted on Time-MMD through MM-TSFlib demonstrate significant performance enhancements by extending unimodal TSF to multimodality, evidenced by over 15% mean squared error reduction in general, and up to 40% in domains with rich textual data. More importantly, our datasets and library revolutionize broader applications, impacts, research topics to advance TSA. The dataset is available at https://github.com/AdityaLab/Time-MMD.

Figures (10)

• Figure 1: Overview of the Time-MMD construction. We first construct numerical data, then construct textual data from search and report sources with LLM preprocessing targeted at the numerical data, and finally annotate the data with binary timestamps to support various downstream tasks.
• Figure 2: Visualization of Time-MMD, highlighting distinct characteristics across different domains.
• Figure 3: Visualization of relevant report (a, left) and search (b, right) counts in Time-MMD over time. Text counts from both reports and searches increase over time. Domains receiving more attention, such as the economy, contain more available relevant text data..
• Figure 4: Word cloud visualization for influenza patient proportion from the health domain. The discarded texts are identified by the LLM as irrelevant to the target variable. The results validate the effectiveness of LLM preprocessing.
• Figure 5: Overall structure of the MM-TSFlib. MM-TSFlib uses a model-agnostic multimodal integration framework that independently models numerical and textual series within an end-to-end training manner. MM-TSFlib slightly increases the number of trainable parameters, balancing effectiveness and efficiency.