Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TSRM: A Lightweight Temporal Feature Encoding Architecture for Time Series Forecasting and Imputation

Robert Leppich, Michael Stenger, Daniel Grillmeyer, Vanessa Borst, Samuel Kounev

TL;DR

TSRM introduces a lightweight, Transformer-encoder-inspired architecture for multivariate time series forecasting and imputation. It uses stackable Encoding Layers where each EL combines a Representation Layer built from multi-scale 1D CNNs and a Merge Layer to restore dimensions, with two residual blocks incorporating self-attention. A channel-independent variant (TSRM) and an inter-feature correlated variant (TSRM_IFC) enable both per-channel learning and cross-feature interactions, achieving state-of-the-art or competitive results on seven benchmarks while maintaining a small parameter footprint. The model provides explainability by mapping attention weights back to inputs and demonstrates robustness across forecasting and imputation tasks, supported by ablations and complexity analysis. The work offers a practical, extensible approach with open-source code for reproducibility and further research.

Abstract

We introduce a temporal feature encoding architecture called Time Series Representation Model (TSRM) for multivariate time series forecasting and imputation. The architecture is structured around CNN-based representation layers, each dedicated to an independent representation learning task and designed to capture diverse temporal patterns, followed by an attention-based feature extraction layer and a merge layer, designed to aggregate extracted features. The architecture is fundamentally based on a configuration that is inspired by a Transformer encoder, with self-attention mechanisms at its core. The TSRM architecture outperforms state-of-the-art approaches on most of the seven established benchmark datasets considered in our empirical evaluation for both forecasting and imputation tasks. At the same time, it significantly reduces complexity in the form of learnable parameters. The source code is available at https://github.com/RobertLeppich/TSRM.

TSRM: A Lightweight Temporal Feature Encoding Architecture for Time Series Forecasting and Imputation

TL;DR

TSRM introduces a lightweight, Transformer-encoder-inspired architecture for multivariate time series forecasting and imputation. It uses stackable Encoding Layers where each EL combines a Representation Layer built from multi-scale 1D CNNs and a Merge Layer to restore dimensions, with two residual blocks incorporating self-attention. A channel-independent variant (TSRM) and an inter-feature correlated variant (TSRM_IFC) enable both per-channel learning and cross-feature interactions, achieving state-of-the-art or competitive results on seven benchmarks while maintaining a small parameter footprint. The model provides explainability by mapping attention weights back to inputs and demonstrates robustness across forecasting and imputation tasks, supported by ablations and complexity analysis. The work offers a practical, extensible approach with open-source code for reproducibility and further research.

Abstract

We introduce a temporal feature encoding architecture called Time Series Representation Model (TSRM) for multivariate time series forecasting and imputation. The architecture is structured around CNN-based representation layers, each dedicated to an independent representation learning task and designed to capture diverse temporal patterns, followed by an attention-based feature extraction layer and a merge layer, designed to aggregate extracted features. The architecture is fundamentally based on a configuration that is inspired by a Transformer encoder, with self-attention mechanisms at its core. The TSRM architecture outperforms state-of-the-art approaches on most of the seven established benchmark datasets considered in our empirical evaluation for both forecasting and imputation tasks. At the same time, it significantly reduces complexity in the form of learnable parameters. The source code is available at https://github.com/RobertLeppich/TSRM.

Paper Structure

This paper contains 27 sections, 4 equations, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Model Architecture
  4. Encoding Layer (EL)
  5. Representation Layer (RL)
  6. Merge Layer
  7. Middle Encoding Blocks
  8. Explainability
  9. Experiments
  10. Experimental Setup
  11. Long-Term Time Series Forecasting
  12. Results
  13. Imputation
  14. Results
  15. Complexity Analysis
  16. ...and 12 more sections

Figures (3)

  • Figure 1: Illustration of the proposed Time Series Representation Model (TSRM) framework, primarily composed of $N$ encoding layers (ELs) (upper section in blue), accompanied by the representation layer (RL) (left, in green) and merge layer (ML) (right, in orange).
  • Figure 2: Ablation study results for the architecture variations (bottom) and sensitivity study for the hyperparameter $N$ (top) on the datasets Weather and ETTh1.
  • Figure 3: Highlighted attention weights during an ETTh1 forecasting task for all 3 ELs, starting with the first EL at the top and concluding with the the combined version over all EL at the bottom.