Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Set-Sequence Model for Time Series

Elliot L. Epstein, Apaar Sadhwani, Kay Giesecke

TL;DR

The paper introduces Set-Sequence, a scalable architecture for predicting time-series outcomes across large populations of exchangeable units. It combines a permutation-invariant Set module to learn a cross-sectional summary at each time step with a per-unit Sequence backbone that models temporal dynamics conditioned on this summary, enabling processing with a variable number of units and unaligned sequences. The approach achieves linear cross-sectional complexity, demonstrates strong performance on synthetic contagion tasks, equity portfolio construction, and mortgage risk prediction, and provides interpretable cross-sectional summaries that track latent factors. These results highlight the practical usefulness of exploiting exchangeability to jointly model large populations without handcrafted cross-sectional features, with broad implications for finance and other domains with many interacting time-series units.

Abstract

Many prediction problems across science and engineering, especially in finance and economics, involve large cross-sections of individual time series, where each unit (e.g., a loan, stock, or customer) is driven by unit-level features and latent cross-sectional dynamics. While sequence models have advanced per-unit temporal prediction, capturing cross-sectional effects often still relies on hand-crafted summary features. We propose Set-Sequence, a model that learns cross-sectional structure directly, enhancing expressivity and eliminating manual feature engineering. At each time step, a permutation-invariant Set module summarizes the unit set; a Sequence module then models each unit's dynamics conditioned on both its features and the learned summary. The architecture accommodates unaligned series, supports varying numbers of units at inference, integrates with standard sequence backbones (e.g., Transformers), and scales linearly in cross-sectional size. Across a synthetic contagion task and two large-scale real-world applications, equity portfolio optimization and loan risk prediction, Set-Sequence significantly outperforms strong baselines, delivering higher Sharpe ratios, improved AUCs, and interpretable cross-sectional summaries.

A Set-Sequence Model for Time Series

TL;DR

The paper introduces Set-Sequence, a scalable architecture for predicting time-series outcomes across large populations of exchangeable units. It combines a permutation-invariant Set module to learn a cross-sectional summary at each time step with a per-unit Sequence backbone that models temporal dynamics conditioned on this summary, enabling processing with a variable number of units and unaligned sequences. The approach achieves linear cross-sectional complexity, demonstrates strong performance on synthetic contagion tasks, equity portfolio construction, and mortgage risk prediction, and provides interpretable cross-sectional summaries that track latent factors. These results highlight the practical usefulness of exploiting exchangeability to jointly model large populations without handcrafted cross-sectional features, with broad implications for finance and other domains with many interacting time-series units.

Abstract

Many prediction problems across science and engineering, especially in finance and economics, involve large cross-sections of individual time series, where each unit (e.g., a loan, stock, or customer) is driven by unit-level features and latent cross-sectional dynamics. While sequence models have advanced per-unit temporal prediction, capturing cross-sectional effects often still relies on hand-crafted summary features. We propose Set-Sequence, a model that learns cross-sectional structure directly, enhancing expressivity and eliminating manual feature engineering. At each time step, a permutation-invariant Set module summarizes the unit set; a Sequence module then models each unit's dynamics conditioned on both its features and the learned summary. The architecture accommodates unaligned series, supports varying numbers of units at inference, integrates with standard sequence backbones (e.g., Transformers), and scales linearly in cross-sectional size. Across a synthetic contagion task and two large-scale real-world applications, equity portfolio optimization and loan risk prediction, Set-Sequence significantly outperforms strong baselines, delivering higher Sharpe ratios, improved AUCs, and interpretable cross-sectional summaries.
Paper Structure (66 sections, 3 theorems, 28 equations, 18 figures, 12 tables)

This paper contains 66 sections, 3 theorems, 28 equations, 18 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Literature
  3. Set-Sequence Model
  4. Set-Sequence Layer
  5. Set-Sequence Model
  6. Alternative Layer Formulations
  7. Expressivity under exchangeability
  8. Task Selection and Baselines
  9. Task Selection
  10. General and Domain-Specific Baselines
  11. Synthetic Task
  12. Comparison with Sequence Models
  13. Model Ablations
  14. Generalization Across Number of Input Units
  15. Application: Equity Portfolio Construction
  16. ...and 51 more sections

Key Result

Proposition 1

Let $M$ be the number of units, $d$ the number of features per unit, and $T$ the sequence length. At each time step $t\in\{1,\dots,T\}$ we hold $M$ vectors $x^{(i)}_t\in\mathbb{R}^{d}$. Let $C_{\mathrm{seq}}(T,w)$ denote the cost of one temporal layer on a length-$T$ sequence of width $w$. The forwa MHA--Seq applies cross-sectional self-attention across the $M$ units, whereas the Naïve MHA--Seq in

Figures (18)

  • Figure 1: The Set-Sequence model. The Set model estimates a cross-sectional summary at each time, in linear complexity over the number of units. This augments the original features of each unit and the Sequence model consumes this augmented series to make predictions for each unit independently. Note that the Set model has a look-back of $L$ time periods, where $L\geq 1$ is a model parameter. The number of units $M$ and time periods $T$ may vary at inference.
  • Figure 2: Comparison between the Set-Sequence model, MHA-Sequence model, and the Kalman Filter, by considering the AUC, $R^2$, and Correlation, each for the absorbing (rare) state. The Set-Sequence model reaches near the oracle-like Kalman Filter model performance across the full range of observed units for inference, for all the considered metrics.
  • Figure 3: AUC gain (Set-Sequence minus baseline dlmr) for each transition. Deeper green indicates a larger Set-Sequence advantage; red indicates the baseline performed better. Larger circles indicate more common transitions, with size proportional to the log of their frequency. Only transitions occurring at least 10 times are included. In parentheses is the standard deviation over 10 random subsets of the evaluation set.
  • Figure 4: The performance for the MHA-Seq model depending on the training sampling method. $\delta_{k}$ denotes that all samples have input k units. The method are compute adjusted so that the number of epochs are scaled up to 400 (vs 40) for the case with input size 100, and scaled up to 4000 for input size 10.
  • Figure 5: Out-of-sample annualized performance metrics from 2002 to 2016 for the stock portfolio construction task. All CNN-Transformer models are based on the IPCA method described in guijarroordonez2022deeplearningstatisticalarbitrage. For the Set-Sequence model the results are the mean over 5 training initialization seeds. For the Sharpe we also report its standard deviation over the seeds.
  • ...and 13 more figures

Theorems & Definitions (5)

  • Proposition 1: Forward time for one cross-sectional layer
  • Proposition 2: Expressivity of Set Module
  • proof
  • Corollary 1: Plugging in standard temporal self-attention
  • proof