Abstain Mask Retain Core: Time Series Prediction by Adaptive Masking Loss with Representation Consistency

TL;DR

This work addresses time series forecasting by challenging the long-sequence information gain hypothesis and showing that truncating history can improve accuracy due to redundancy in learned features. It introduces Adaptive Masking Loss with Representation Consistency (AMRC), combining AML to dynamically mask informative temporal segments and ESP to align embedding and output spaces under an IB-inspired framework. Theoretical grounding via the Information Bottleneck and extensive experiments across seven datasets and multiple backbones demonstrate architecture-agnostic gains and synergy between AML and ESP. The approach offers a practical, plug-in training framework that reduces redundancy, enhances generalization, and supports robust, efficient forecasting in diverse domains.

Abstract

Time series forecasting plays a pivotal role in critical domains such as energy management and financial markets. Although deep learning-based approaches (e.g., MLP, RNN, Transformer) have achieved remarkable progress, the prevailing "long-sequence information gain hypothesis" exhibits inherent limitations. Through systematic experimentation, this study reveals a counterintuitive phenomenon: appropriately truncating historical data can paradoxically enhance prediction accuracy, indicating that existing models learn substantial redundant features (e.g., noise or irrelevant fluctuations) during training, thereby compromising effective signal extraction. Building upon information bottleneck theory, we propose an innovative solution termed Adaptive Masking Loss with Representation Consistency (AMRC), which features two core components: 1) Dynamic masking loss, which adaptively identified highly discriminative temporal segments to guide gradient descent during model training; 2) Representation consistency constraint, which stabilized the mapping relationships among inputs, labels, and predictions. Experimental results demonstrate that AMRC effectively suppresses redundant feature learning while significantly improving model performance. This work not only challenges conventional assumptions in temporal modeling but also provides novel theoretical insights and methodological breakthroughs for developing efficient and robust forecasting models.

Figures (12)

• Figure 1: Illustration of the effect of AMRC method. Without regularization, the model tends to overfit redundant input features, leading to suboptimal convergence. By suppressing redundant input features, AMRC restructures the optimization landscape, promoting more efficient representation learning and facilitating better convergence.
• Figure 2: Embedding Distributions and Masking Effects of Our Method.
• Figure 3: Overview of the Adaptive Masking Loss (AML) framework. The upper half illustrates how the optimal mask length $K^*$ is selected by evaluating prediction losses over sampled masks. A weighting coefficient $\beta$ is computed based on the gain over the unmasked loss. The lower half shows the AML loss, calculated as the sum of representation differences between the original input and the $K^*$ masked input across embedding, backbone, and predictor layers.
• Figure : (a) SOFTS in ETTh1
• Figure : (a) SOFTS in ETTh1