Enhancing Time Series Forecasting via Logic-Inspired Regularization

TL;DR

The paper targets a key limitation of Transformer-based TSF: treating all token dependencies equally, which hurts performance when dependencies vary by forecasting scenario. It introduces Attention Logic Regularization (Attn-L-Reg), a plug-in sparsity regularizer grounded in a logic-inspired notion of atomic token representations to enforce minimal, effective dependencies. The authors provide a theoretical generalization bound showing the benefit of L1 regularization on attention and demonstrate strong empirical gains across six real-world datasets, with analyses showing reduced redundancy in attention. This approach yields a practical, model-agnostic improvement for TSF that enhances generalization and interpretability by focusing on the most informative token dependencies.

Abstract

Time series forecasting (TSF) plays a crucial role in many applications. Transformer-based methods are one of the mainstream techniques for TSF. Existing methods treat all token dependencies equally. However, we find that the effectiveness of token dependencies varies across different forecasting scenarios, and existing methods ignore these differences, which affects their performance. This raises two issues: (1) What are effective token dependencies? (2) How can we learn effective dependencies? From a logical perspective, we align Transformer-based TSF methods with the logical framework and define effective token dependencies as those that ensure the tokens as atomic formulas (Issue 1). We then align the learning process of Transformer methods with the process of obtaining atomic formulas in logic, which inspires us to design a method for learning these effective dependencies (Issue 2). Specifically, we propose Attention Logic Regularization (Attn-L-Reg), a plug-and-play method that guides the model to use fewer but more effective dependencies by making the attention map sparse, thereby ensuring the tokens as atomic formulas and improving prediction performance. Extensive experiments and theoretical analysis confirm the effectiveness of Attn-L-Reg.

Figures (7)

• Figure 1: Example of time series forecasting. The yellow point represents the first time point to be predicted, which does not need to span a cycle. The red point represents the last point to be predicted, which requires spanning approximately four cycles.
• Figure 2: The results of the empirical analysis. The used algorithm, dataset, and predicted point position (first or last) are shown at the top of each image. The x and y axes represent token indices. The color in the $i$-th row and $j$-th column indicates the change in model performance after removing the dependency between the $i$-th and $j$-th tokens: purple for improvement, and beige for decline. Due to space limitations, only the dependencies between a maximum of 50 tokens are shown in the figure.
• Figure 3: The overall framework of the proposed method.
• Figure 4: The visualized redundant dependencies. The method, dataset, and MSE are shown at the top of each image. The colors follow Fig.\ref{['fig_motivation']}. Due to the space limitations, only dependencies among 50 tokens are displayed in this figure.
• Figure 5: Visualization of input-96-predict-96 results on the Traffic dataset.

Theorems & Definitions (15)

• Definition 4.1: Logic andreka2017universal
• Definition 4.2: Logic in TSF
• Definition 4.3: Atomic Formula in Transformer-based TSF
• Definition 4.4: Effective Token Dependencies
• Theorem 6.1: Generalization Bound for Transformer-based TSF
• Theorem 6.2: Better Generalization Error Upper Bound
• Definition 4.1: Logic
• Definition 4.2: Atomic Formula and Composite Formula
• Definition 5.1: Generalization Error in Regression Problem
• Definition 5.2: Empirical Error in Regression Problem