Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Covariance-Aware Transformers for Quadratic Programming and Decision Making

Kutay Tire, Yufan Zhang, Ege Onur Taga, Samet Oymak

TL;DR

The paper investigates whether transformer architectures can function as general-purpose solvers for convex quadratic programs and leverages this capability to improve decision making when second-order statistics matter. It provides explicit constructions showing that transformers, using linear attention and carefully designed tokens, can emulate gradient descent, primal-dual updates, ISTA, and proximal/projection steps for unconstrained, linearly constrained, and sparse QPs, with convergence guarantees under standard step-size conditions. Building on this theory, the authors introduce Time2Decide, a covariance-aware augmentation that feeds covariance tokens into a time-series foundation model to enable end-to-end forecasting and decision making in a single forward pass, surpassing traditional predict-then-optimize baselines in portfolio optimization under realistic constraints and noise. Empirically, Time2Decide demonstrates strong performance gains over base TSFMs and PtO in many regimes, highlighting the value of explicit second-order statistics in transformer-based decision pipelines. The work positions transformers as both algorithmic emulators for optimization and practical, end-to-end decision makers for complex, covariance-driven problems like portfolio construction.

Abstract

We explore the use of transformers for solving quadratic programs and how this capability benefits decision-making problems that involve covariance matrices. We first show that the linear attention mechanism can provably solve unconstrained QPs by tokenizing the matrix variables (e.g.~$A$ of the objective $\frac{1}{2}x^\top Ax+b^\top x$) row-by-row and emulating gradient descent iterations. Furthermore, by incorporating MLPs, a transformer block can solve (i) $\ell_1$-penalized QPs by emulating iterative soft-thresholding and (ii) $\ell_1$-constrained QPs when equipped with an additional feedback loop. Our theory motivates us to introduce Time2Decide: a generic method that enhances a time series foundation model (TSFM) by explicitly feeding the covariance matrix between the variates. We empirically find that Time2Decide uniformly outperforms the base TSFM model for the classical portfolio optimization problem that admits an $\ell_1$-constrained QP formulation. Remarkably, Time2Decide also outperforms the classical "Predict-then-Optimize (PtO)" procedure, where we first forecast the returns and then explicitly solve a constrained QP, in suitable settings. Our results demonstrate that transformers benefit from explicit use of second-order statistics, and this can enable them to effectively solve complex decision-making problems, like portfolio construction, in one forward pass.

Covariance-Aware Transformers for Quadratic Programming and Decision Making

TL;DR

The paper investigates whether transformer architectures can function as general-purpose solvers for convex quadratic programs and leverages this capability to improve decision making when second-order statistics matter. It provides explicit constructions showing that transformers, using linear attention and carefully designed tokens, can emulate gradient descent, primal-dual updates, ISTA, and proximal/projection steps for unconstrained, linearly constrained, and sparse QPs, with convergence guarantees under standard step-size conditions. Building on this theory, the authors introduce Time2Decide, a covariance-aware augmentation that feeds covariance tokens into a time-series foundation model to enable end-to-end forecasting and decision making in a single forward pass, surpassing traditional predict-then-optimize baselines in portfolio optimization under realistic constraints and noise. Empirically, Time2Decide demonstrates strong performance gains over base TSFMs and PtO in many regimes, highlighting the value of explicit second-order statistics in transformer-based decision pipelines. The work positions transformers as both algorithmic emulators for optimization and practical, end-to-end decision makers for complex, covariance-driven problems like portfolio construction.

Abstract

We explore the use of transformers for solving quadratic programs and how this capability benefits decision-making problems that involve covariance matrices. We first show that the linear attention mechanism can provably solve unconstrained QPs by tokenizing the matrix variables (e.g.~ of the objective ) row-by-row and emulating gradient descent iterations. Furthermore, by incorporating MLPs, a transformer block can solve (i) -penalized QPs by emulating iterative soft-thresholding and (ii) -constrained QPs when equipped with an additional feedback loop. Our theory motivates us to introduce Time2Decide: a generic method that enhances a time series foundation model (TSFM) by explicitly feeding the covariance matrix between the variates. We empirically find that Time2Decide uniformly outperforms the base TSFM model for the classical portfolio optimization problem that admits an -constrained QP formulation. Remarkably, Time2Decide also outperforms the classical "Predict-then-Optimize (PtO)" procedure, where we first forecast the returns and then explicitly solve a constrained QP, in suitable settings. Our results demonstrate that transformers benefit from explicit use of second-order statistics, and this can enable them to effectively solve complex decision-making problems, like portfolio construction, in one forward pass.
Paper Structure (73 sections, 6 theorems, 67 equations, 10 figures, 47 tables)

This paper contains 73 sections, 6 theorems, 67 equations, 10 figures, 47 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neural Networks as Optimizers.
  4. Multivariate Time-Series Transformers.
  5. Theoretical Results
  6. Problem Setup
  7. Notation.
  8. Quadratic programs.
  9. Algorithms to be emulated.
  10. Unconstrained QP via Single-Block Linear Attention
  11. Linearly Constrained QP via Single Macro-Block Linear Attention
  12. Sparse QPs via Single-Block Linear Attention: L1-Regularized and L1-Constrained
  13. Neural QP Solver Experiments
  14. Experimental Design
  15. Model architectures and metrics.
  16. ...and 58 more sections

Key Result

Proposition 3.1

A single linear-attention head that attends from the $x$-row to the $A$-rows, followed by a residual on the $x$-row, realizes the update $x_{k+1}=x_k-\gamma(Ax_k+b)$. Hence, for any $0<\gamma<2/L$, the iterates converge linearly to $x^\star=-A^{-1}b$.

Figures (10)

  • Figure 1: Overview of portfolio decision-making pipelines.(a) Predict-then-Optimize (PtO): historical multivariate returns are tokenized into patches and fed to TimePFN to predict next-step returns $\hat{r}_t$. A QP solver then takes these predictions to compute the allocation $s_t$ under simplex and $\ell_1$ rebalancing constraints (b) Time2Decide: unifies both stages in a single forward pass. We first derive learned covariance tokens from $\hat{\Sigma}_t$ by applying an MLP and prepend them to the return and allocation patch tokens. TSFM with joint self-attention processes this full sequence to directly output both $\hat{r}_t$ and $\hat{s}_t$. The final feasibility projection enforces the portfolio constraints to yield $s_t$, enabling an end-to-end, covariance-aware decision without a separate optimization step.
  • Figure 2: Convergence on a semilog-$y$ scale. Each panel compares a standard algorithm (orange) with our transformer-based construction (blue): (a) GD for the unconstrained case (U), (b) ISTA for the regularized case (R), and (c) PGD for the constrained case (C). The curves nearly overlap across layers (depth $=$ iterations), showing that the transformer mirrors the reference methods.
  • Figure 3: Test-set NMSE histogram for the LinearTransformer (8 layers, 2 heads). Dashed lines mark the mean and median; the dotted line marks the 95th percentile. Errors are concentrated near zero (median $\approx$ 0.011; 95% $\approx$ 0.095).
  • Figure 4: Error distributions for the best LinearTransformer configuration ($n=10, m=3$, layers=8, heads=2).
  • Figure 5: Error distributions for the best SoftmaxTransformer configuration ($n=10, m=3$, layers=16, heads=8).
  • ...and 5 more figures

Theorems & Definitions (15)

  • Proposition 3.1
  • proof : Construction
  • Proposition 3.2
  • proof : Construction
  • Proposition 3.3
  • proof : Construction
  • Proposition 3.4
  • proof : Construction
  • Proposition 5.1: Informal: PtO is suboptimal under estimation noise
  • proof : Construction
  • ...and 5 more