TimeSieve: Extracting Temporal Dynamics through Information Bottlenecks

TL;DR

TimeSieve tackles the challenges of manual hyperparameter tuning and distinguishing signal from redundant features in time-series forecasting by fusing wavelet-based multi-scale decomposition with an information bottleneck module. It preprocesses data with a Wavelet Decomposition Block, filters redundancy via an Information Filtering and Compression Block guided by the Information Bottleneck objective, and reconstructs with a Wavelet Reconstruction Block before forecasting with an MLP; the IB objective balances maximizing mutual information with the target and minimizing information about the input, formalized as $ \max I(y; z) - \min I(x; z)$, with the IFCB loss incorporating $D_{KL}$ terms. The approach achieves state-of-the-art performance on approximately 70% of benchmarks, especially on datasets with strong seasonal redundancy, and demonstrates robust generalization across diverse domains. Overall, TimeSieve offers a practical, parameter-efficient framework for reliable, scalable time-series forecasting that leverages multi-scale structure and information-theoretic filtering.

Abstract

Time series forecasting has become an increasingly popular research area due to its critical applications in various real-world domains such as traffic management, weather prediction, and financial analysis. Despite significant advancements, existing models face notable challenges, including the necessity of manual hyperparameter tuning for different datasets, and difficulty in effectively distinguishing signal from redundant features in data characterized by strong seasonality. These issues hinder the generalization and practical application of time series forecasting models. To solve this issues, we propose an innovative time series forecasting model TimeSieve designed to address these challenges. Our approach employs wavelet transforms to preprocess time series data, effectively capturing multi-scale features without the need for additional parameters or manual hyperparameter tuning. Additionally, we introduce the information bottleneck theory that filters out redundant features from both detail and approximation coefficients, retaining only the most predictive information. This combination reduces significantly improves the model's accuracy. Extensive experiments demonstrate that our model outperforms existing state-of-the-art methods on 70% of the datasets, achieving higher predictive accuracy and better generalization across diverse datasets. Our results validate the effectiveness of our approach in addressing the key challenges in time series forecasting, paving the way for more reliable and efficient predictive models in practical applications. The code for our model is available at https://github.com/xll0328/TimeSieve.

Figures (4)

• Figure 1: Overview of the TimeSieve Framework. The upper part illustrates the overall architecture of TimeSieve, which consists of WDB, WRB and IFCB. The input time series data, $T \times C$, is decomposed by WDB into coefficients $\pi_a$ and $\pi_d$, which are processed by IFCB. The WRB then reconstructs the data, followed by the predictor generating the corresponding forecast steps. The lower part provides a detailed visualization of the data flow within WDB and IFCB, with further visualization of IFCB available in Figure \ref{['fig:IFCB']}.
• Figure 2: Overview of IFCB: This module consists of a reparameterization step and deep neural networks. Initially, the standard deviation and mean ($\mu_z, \Sigma_z$) of the data are computed. The reparameterization is then performed using the equation $z = \mu_z + \Sigma_z \times \epsilon$. The input coefficients are processed through the first neural network to obtain $\mu_z$ and $\Sigma_z$. The reparameterized $z$ is then fed into the second neural network to produce the reconstructed coefficients $\hat{\pi_i}$.
• Figure 3: The figure depicts the evolution of the mean (MU) and standard deviation (STD) within the information bottleneck as model iterations progress, with MSE as the metric, for a prediction length of 48 on the Exchange dataset.
• Figure 4: The figure illustrates the changes in the mutual information related to the input $I(X;T)$ and the output $\beta I(T;Y)$ throughout model iterations, measured by MSE, for a prediction length of 48 on the Exchange dataset.