Metadata Matters for Time Series: Informative Forecasting with Transformers

TL;DR

A Metadata-informed Time Series Transformer (MetaTST), which incorporates multiple levels of context-specific metadata into Transformer forecasting models to enable informative time series forecasting and achieves state-of-the-art performance on widely acknowledged short- and long-term forecasting benchmarks.

Abstract

Time series forecasting is prevalent in extensive real-world applications, such as financial analysis and energy planning. Previous studies primarily focus on time series modality, endeavoring to capture the intricate variations and dependencies inherent in time series. Beyond numerical time series data, we notice that metadata (e.g.~dataset and variate descriptions) also carries valuable information essential for forecasting, which can be used to identify the application scenario and provide more interpretable knowledge than digit sequences. Inspired by this observation, we propose a Metadata-informed Time Series Transformer (MetaTST), which incorporates multiple levels of context-specific metadata into Transformer forecasting models to enable informative time series forecasting. To tackle the unstructured nature of metadata, MetaTST formalizes them into natural languages by pre-designed templates and leverages large language models (LLMs) to encode these texts into metadata tokens as a supplement to classic series tokens, resulting in an informative embedding. Further, a Transformer encoder is employed to communicate series and metadata tokens, which can extend series representations by metadata information for more accurate forecasting. This design also allows the model to adaptively learn context-specific patterns across various scenarios, which is particularly effective in handling large-scale, diverse-scenario forecasting tasks. Experimentally, MetaTST achieves state-of-the-art compared to advanced time series models and LLM-based methods on widely acknowledged short- and long-term forecasting benchmarks, covering both single-dataset individual and multi-dataset joint training settings.

Figures (13)

• Figure 1: Comparison on different time series forecasting paradigms. (a) Canonical time series forecasting without metadata. (b) Metadata-informed time series forecasting and (c) MetaTST utilizes more informative inputs, especially context-specific metadata, to achieve highly certain forecasts.
• Figure 2: The overall design of MetaTST, which integrates endogenous series, exogenous series, and context-specific textual metadata to enable informative time series forecasting with Transformers.
• Figure 3: Different aggregating methods to transform word-level token sequences to global-level.
• Figure 4: Ablation studies of MetaTST with various types informative forecasting, covering individual and joint training strategy in long- and short-term forecasting tasks. More details are in Appendix \ref{['app:additional_ablation']}.
• Figure 5: (a) Performance comparison on different LLMs as the metadata encoder and (b) Representation visualization of metadata on short-term (left) and long-term (right) joint training settings.