Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Extended Deep Adaptive Input Normalization for Preprocessing Time Series Data for Neural Networks

Marcus A. K. September, Francesco Sanna Passino, Leonie Goldmann, Anton Hinel

TL;DR

The paper tackles the challenge of preprocessing multivariate time series for neural networks by introducing EDAIN, an end-to-end adaptive input normalization layer that jointly learns to mitigate outliers, skewness, and multi-modality. EDAIN comprises four sublayers—outlier mitigation, shift, scale, and a power transform—and exists in global-aware and local-aware forms, with an additional KL-optimised unsupervised variant EDAIN-KL. The approach outperforms static preprocessing and existing adaptive methods across synthetic data, the AmEx default dataset, and the FI-2010 LOB benchmark, with notable gains on unimodal versus multimodal datasets depending on the configuration. The work also provides an open-source implementation and demonstrates practical impact for finance and credit risk tasks, enabling more effective preprocessing in time-series neural networks.

Abstract

Data preprocessing is a crucial part of any machine learning pipeline, and it can have a significant impact on both performance and training efficiency. This is especially evident when using deep neural networks for time series prediction and classification: real-world time series data often exhibit irregularities such as multi-modality, skewness and outliers, and the model performance can degrade rapidly if these characteristics are not adequately addressed. In this work, we propose the EDAIN (Extended Deep Adaptive Input Normalization) layer, a novel adaptive neural layer that learns how to appropriately normalize irregular time series data for a given task in an end-to-end fashion, instead of using a fixed normalization scheme. This is achieved by optimizing its unknown parameters simultaneously with the deep neural network using back-propagation. Our experiments, conducted using synthetic data, a credit default prediction dataset, and a large-scale limit order book benchmark dataset, demonstrate the superior performance of the EDAIN layer when compared to conventional normalization methods and existing adaptive time series preprocessing layers.

Extended Deep Adaptive Input Normalization for Preprocessing Time Series Data for Neural Networks

TL;DR

The paper tackles the challenge of preprocessing multivariate time series for neural networks by introducing EDAIN, an end-to-end adaptive input normalization layer that jointly learns to mitigate outliers, skewness, and multi-modality. EDAIN comprises four sublayers—outlier mitigation, shift, scale, and a power transform—and exists in global-aware and local-aware forms, with an additional KL-optimised unsupervised variant EDAIN-KL. The approach outperforms static preprocessing and existing adaptive methods across synthetic data, the AmEx default dataset, and the FI-2010 LOB benchmark, with notable gains on unimodal versus multimodal datasets depending on the configuration. The work also provides an open-source implementation and demonstrates practical impact for finance and credit risk tasks, enabling more effective preprocessing in time-series neural networks.

Abstract

Data preprocessing is a crucial part of any machine learning pipeline, and it can have a significant impact on both performance and training efficiency. This is especially evident when using deep neural networks for time series prediction and classification: real-world time series data often exhibit irregularities such as multi-modality, skewness and outliers, and the model performance can degrade rapidly if these characteristics are not adequately addressed. In this work, we propose the EDAIN (Extended Deep Adaptive Input Normalization) layer, a novel adaptive neural layer that learns how to appropriately normalize irregular time series data for a given task in an end-to-end fashion, instead of using a fixed normalization scheme. This is achieved by optimizing its unknown parameters simultaneously with the deep neural network using back-propagation. Our experiments, conducted using synthetic data, a credit default prediction dataset, and a large-scale limit order book benchmark dataset, demonstrate the superior performance of the EDAIN layer when compared to conventional normalization methods and existing adaptive time series preprocessing layers.
Paper Structure (59 sections, 1 theorem, 52 equations, 5 figures, 4 tables)

This paper contains 59 sections, 1 theorem, 52 equations, 5 figures, 4 tables.

Table of Contents

  1. INTRODUCTION
  2. Preprocessing multivariate time series
  3. Contributions
  4. RELATED WORK
  5. EXTENDED DEEP ADAPTIVE INPUT NORMALIZATION
  6. Outlier mitigation sublayer.
  7. Shift and scale sublayers.
  8. Power transform sublayer.
  9. Global- and local-aware normalization
  10. Order preservation.
  11. Optimising the EDAIN layer
  12. EDAIN-KL
  13. EXPERIMENTAL EVALUATION
  14. Synthetic Datasets
  15. Default Prediction Dataset
  16. ...and 44 more sections

Key Result

Proposition B.1

Let $\mathbf{g}_1,\dots, \mathbf{g}_n:\mathbb{R}^d \rightarrow {\mathbb{R}^d}$ all be bijective functions, and consider the composition of these functions, $\mathbf{g}=\mathbf{g}_n \circ \mathbf{g}_{n-1} \cdots \circ \mathbf{g}_1$. Then, $\mathbf{g}$ is a bijective function with inverse and the log of the absolute value of the determinant of the Jacobian is given by

Figures (5)

  • Figure 1: Architecture of the proposed EDAIN (Extended Deep Adaptive Input Normalization) layer. The layout and color choices of the diagram are based on Figure 1 from dain.
  • Figure 2: Visual comparison of the local- and global-aware versions of adaptive preprocessing schemes.
  • Figure 3: Histogram across timesteps $t=1,\dots,T$ of the $d=3$ predictor variables from the synthetic data.
  • Figure 4: BCE cross-validation loss across different folds in the Amex default prediction dataset.
  • Figure 5: High-level overview of the proposed synthetic data generation algorithm.

Theorems & Definitions (1)

  • Proposition B.1