Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Incorporating Exponential Smoothing into MLP: A Simple but Effective Sequence Model

Jiqun Chu, Zuoquan Lin

TL;DR

This work investigates long-range sequence modeling by probing whether the success of structured state-space approaches like S4 stems from their complex parameterization or from the broader use of state-space models. The authors start from a discrete ETS, a simple SSM, and integrate a parameterized Complex Exponential Smoothing module into an element-wise MLP to create the ETSMLP, with minimal parameter overhead. They introduce stability-enhancing techniques such as an exponential parameterization \lambda' = \log\log \lambda, a constraint ensuring \lambda^{\alpha} stays within the unit disk, and a gating mechanism, and even extend to bidirectional processing. Empirically, ETSMLP matches S4 on Long Range Arena and remains competitive with transformer encoders on seven NLU datasets, while delivering favorable speed and memory characteristics due to linear scaling with sequence length. Overall, the results suggest that simple SSMs like ETS, when augmented with targeted learnable components, can rival complex SSM-based architectures and offer practical benefits for long-range sequence modeling.

Abstract

Modeling long-range dependencies in sequential data is a crucial step in sequence learning. A recently developed model, the Structured State Space (S4), demonstrated significant effectiveness in modeling long-range sequences. However, It is unclear whether the success of S4 can be attributed to its intricate parameterization and HiPPO initialization or simply due to State Space Models (SSMs). To further investigate the potential of the deep SSMs, we start with exponential smoothing (ETS), a simple SSM, and propose a stacked architecture by directly incorporating it into an element-wise MLP. We augment simple ETS with additional parameters and complex field to reduce the inductive bias. Despite increasing less than 1\% of parameters of element-wise MLP, our models achieve comparable results to S4 on the LRA benchmark.

Incorporating Exponential Smoothing into MLP: A Simple but Effective Sequence Model

TL;DR

This work investigates long-range sequence modeling by probing whether the success of structured state-space approaches like S4 stems from their complex parameterization or from the broader use of state-space models. The authors start from a discrete ETS, a simple SSM, and integrate a parameterized Complex Exponential Smoothing module into an element-wise MLP to create the ETSMLP, with minimal parameter overhead. They introduce stability-enhancing techniques such as an exponential parameterization \lambda' = \log\log \lambda, a constraint ensuring \lambda^{\alpha} stays within the unit disk, and a gating mechanism, and even extend to bidirectional processing. Empirically, ETSMLP matches S4 on Long Range Arena and remains competitive with transformer encoders on seven NLU datasets, while delivering favorable speed and memory characteristics due to linear scaling with sequence length. Overall, the results suggest that simple SSMs like ETS, when augmented with targeted learnable components, can rival complex SSM-based architectures and offer practical benefits for long-range sequence modeling.

Abstract

Modeling long-range dependencies in sequential data is a crucial step in sequence learning. A recently developed model, the Structured State Space (S4), demonstrated significant effectiveness in modeling long-range sequences. However, It is unclear whether the success of S4 can be attributed to its intricate parameterization and HiPPO initialization or simply due to State Space Models (SSMs). To further investigate the potential of the deep SSMs, we start with exponential smoothing (ETS), a simple SSM, and propose a stacked architecture by directly incorporating it into an element-wise MLP. We augment simple ETS with additional parameters and complex field to reduce the inductive bias. Despite increasing less than 1\% of parameters of element-wise MLP, our models achieve comparable results to S4 on the LRA benchmark.
Paper Structure (19 sections, 2 theorems, 22 equations, 3 figures, 5 tables)

This paper contains 19 sections, 2 theorems, 22 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. State space models
  4. Exponential smoothing
  5. Exponential Smoothing Multi-layer Perceptrons
  6. Complex exponential smoothing module
  7. ETSMLP blocks
  8. Experiments
  9. LRA
  10. NLU
  11. Analysis
  12. Role of damped factors and fields
  13. Initialization of parameters
  14. Efficiency and memory analysis
  15. Related Works
  16. ...and 4 more sections

Key Result

Proposition 1

Let $\lambda \in \mathbb{C}$ be within the interior of the hollow unit disc $D^\circ(0,1)=\{z||z|<1\}/\{(0,0)\}$. We define $\lambda'=\log\log \lambda$ which substitutes $\lambda$ in the equation (eq:cest). If the gradients of $y_t$ satisfy $\sum_{0}^{L} \frac{\mathrm{d} L}{\mathrm{d} y_t} y_t < \in

Figures (3)

  • Figure 1: The relations among SSM, S4, DSS, and ETS. The HiPPO initialization is pointed out in red while the Skew-Hippo initialization is pointed out in orange.
  • Figure 2: Overview of the ETSMLP architecture. The left is the pseudo-code of the complex exponential smoothing (CES) module. The right is the entire architecture with a gate mechanism.
  • Figure 3: A training speed and memory comparison between the transformer and ETSMLP. Both models have approximately 30M parameters, and the batch size remains constant at 1 under all circumstances.

Theorems & Definitions (3)

  • Proposition 1
  • Proposition
  • proof