Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CAPS: Unifying Attention, Recurrence, and Alignment in Transformer-based Time Series Forecasting

Viresh Pati, Yubin Kim, Vinh Pham, Jevon Twitty, Shihao Yang, Jiecheng Lu

TL;DR

CAPS introduces a structured, three-path attention mechanism that decouples global, local, and alignment components in time-series forecasting by combining a Clock-based temporal weighting, SO(2) phase alignment, and diagonal SPD scaling. The three additive paths—Riemann softmax for trend, prefix-product for causal decay, and Clock baseline for temporal priors—enable linear-time computation while preserving rich, content-, interval-, and phase-sensitive representations. Theoretical analysis shows softmax attention inherently couples tokens, while CAPS achieves decoupled aggregation, interval-based decay, and phase-aware matching. Empirically, CAPS outperforms linear-attention baselines and remains competitive with strong baselines across ten datasets, with ablations confirming the three-path design as the source of gains and revealing the favorable performance of the linear variant in time-series forecasting. The work underscores practical impact by offering a lightweight, interpretable, and scalable attention mechanism for diverse forecasting tasks, enabling integration into broader multi-scale architectures.

Abstract

This paper presents $\textbf{CAPS}$ (Clock-weighted Aggregation with Prefix-products and Softmax), a structured attention mechanism for time series forecasting that decouples three distinct temporal structures: global trends, local shocks, and seasonal patterns. Standard softmax attention entangles these through global normalization, while recent recurrent models sacrifice long-term, order-independent selection for order-dependent causal structure. CAPS combines SO(2) rotations for phase alignment with three additive gating paths -- Riemann softmax, prefix-product gates, and a Clock baseline -- within a single attention layer. We introduce the Clock mechanism, a learned temporal weighting that modulates these paths through a shared notion of temporal importance. Experiments on long- and short-term forecasting benchmarks surpass vanilla softmax and linear attention mechanisms and demonstrate competitive performance against seven strong baselines with linear complexity. Our code implementation is available at https://github.com/vireshpati/CAPS-Attention.

CAPS: Unifying Attention, Recurrence, and Alignment in Transformer-based Time Series Forecasting

TL;DR

CAPS introduces a structured, three-path attention mechanism that decouples global, local, and alignment components in time-series forecasting by combining a Clock-based temporal weighting, SO(2) phase alignment, and diagonal SPD scaling. The three additive paths—Riemann softmax for trend, prefix-product for causal decay, and Clock baseline for temporal priors—enable linear-time computation while preserving rich, content-, interval-, and phase-sensitive representations. Theoretical analysis shows softmax attention inherently couples tokens, while CAPS achieves decoupled aggregation, interval-based decay, and phase-aware matching. Empirically, CAPS outperforms linear-attention baselines and remains competitive with strong baselines across ten datasets, with ablations confirming the three-path design as the source of gains and revealing the favorable performance of the linear variant in time-series forecasting. The work underscores practical impact by offering a lightweight, interpretable, and scalable attention mechanism for diverse forecasting tasks, enabling integration into broader multi-scale architectures.

Abstract

This paper presents (Clock-weighted Aggregation with Prefix-products and Softmax), a structured attention mechanism for time series forecasting that decouples three distinct temporal structures: global trends, local shocks, and seasonal patterns. Standard softmax attention entangles these through global normalization, while recent recurrent models sacrifice long-term, order-independent selection for order-dependent causal structure. CAPS combines SO(2) rotations for phase alignment with three additive gating paths -- Riemann softmax, prefix-product gates, and a Clock baseline -- within a single attention layer. We introduce the Clock mechanism, a learned temporal weighting that modulates these paths through a shared notion of temporal importance. Experiments on long- and short-term forecasting benchmarks surpass vanilla softmax and linear attention mechanisms and demonstrate competitive performance against seven strong baselines with linear complexity. Our code implementation is available at https://github.com/vireshpati/CAPS-Attention.
Paper Structure (54 sections, 3 theorems, 19 equations, 2 figures, 7 tables)

This paper contains 54 sections, 3 theorems, 19 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Problem Setup and Notation
  4. Linear Operator View of Sequence Models
  5. Order-Independent Aggregation: Softmax Attention
  6. Order-Dependent Propagation: Prefix Products
  7. Group-Theoretic Decomposition: SO(2) and SPD
  8. Motivating Example: Global---Seasonal---Local Decomposition
  9. Real-world example.
  10. Why neither mechanism suffices alone.
  11. Methodology
  12. CAPS Attention Layer
  13. The Clock Mechanism.
  14. Temporal Alignment.
  15. Path 1: Riemann Softmax.
  16. ...and 39 more sections

Key Result

Proposition 3.1

CAPS with linear attention has complexity $O(Td^2)$; with softmax attention, $O(T^2d)$. The three-path concatenation increases constants by factor $3$ without changing asymptotics.

Figures (2)

  • Figure 1: Decomposition of $y_t$ into global, phase-dependent, and local components. Each requires distinct aggregation properties from the mixing operator $g$.
  • Figure 2: Overview of CAPS: (a) attention kernel, (b) full architecture.

Theorems & Definitions (8)

  • Definition 2.1: SO(2)
  • Definition 2.2: SPD
  • Proposition 3.1: Complexity
  • proof
  • Proposition 3.2: Softmax Couples All Tokens
  • proof
  • Proposition 3.3: CAPS Achieves Decoupled Aggregation
  • proof