Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Higher Order Reduced Rank Regression

Leia Greenberg, Haim Avron

TL;DR

This paper introduces Higher Order Reduced Rank Regression (HORRR), an extension of RRR that leverages multi-linear transformations, and as such is capable of capturing nonlinear interactions in multi-response regression.

Abstract

Reduced Rank Regression (RRR) is a widely used method for multi-response regression. However, RRR assumes a linear relationship between features and responses. While linear models are useful and often provide a good approximation, many real-world problems involve more complex relationships that cannot be adequately captured by simple linear interactions. One way to model such relationships is via multilinear transformations. This paper introduces Higher Order Reduced Rank Regression (HORRR), an extension of RRR that leverages multi-linear transformations, and as such is capable of capturing nonlinear interactions in multi-response regression. HORRR employs tensor representations for the coefficients and a Tucker decomposition to impose multilinear rank constraints as regularization akin to the rank constraints in RRR. Encoding these constraints as a manifold allows us to use Riemannian optimization to solve this HORRR problems. We theoretically and empirically analyze the use of Riemannian optimization for solving HORRR problems.

Higher Order Reduced Rank Regression

TL;DR

This paper introduces Higher Order Reduced Rank Regression (HORRR), an extension of RRR that leverages multi-linear transformations, and as such is capable of capturing nonlinear interactions in multi-response regression.

Abstract

Reduced Rank Regression (RRR) is a widely used method for multi-response regression. However, RRR assumes a linear relationship between features and responses. While linear models are useful and often provide a good approximation, many real-world problems involve more complex relationships that cannot be adequately captured by simple linear interactions. One way to model such relationships is via multilinear transformations. This paper introduces Higher Order Reduced Rank Regression (HORRR), an extension of RRR that leverages multi-linear transformations, and as such is capable of capturing nonlinear interactions in multi-response regression. HORRR employs tensor representations for the coefficients and a Tucker decomposition to impose multilinear rank constraints as regularization akin to the rank constraints in RRR. Encoding these constraints as a manifold allows us to use Riemannian optimization to solve this HORRR problems. We theoretically and empirically analyze the use of Riemannian optimization for solving HORRR problems.

Paper Structure

This paper contains 41 sections, 19 theorems, 167 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reduced Rank Regression.
  4. Tensor Regression.
  5. Riemannian Optimization on Manifolds of Tensors of Low Multilinear Rank.
  6. Preliminaries
  7. Notations and Basic Definitions
  8. Reduced Rank (Linear) Regression
  9. Tensor Apply Operation
  10. Riemannian Optimization
  11. Manifold of Fixed Tucker Rank Tensors
  12. Tangent Space.
  13. Projection on the Tangent Space.
  14. Riemannian Gradient and Riemannian Hessian.
  15. Higher Order SVD (HOSVD) and Retraction.
  16. ...and 26 more sections

Key Result

Lemma 4

Assume that $r_{j}\leq\prod_{i\neq j}r_{i}$ for all $j$. Suppose there exist a dimension $k$ for which $n_{k}=r_{k}$. Then, $\left\{ {\cal G};{\bm{\mathrm{V}}}_{1},\dots,{\bm{\mathrm{V}}}_{d}\right\} =0$ if and only if ${\cal G}=0$ and ${\bm{\mathrm{V}}}_{j}=0$ for $j=1,\dots,d$.

Figures (3)

  • Figure 1: RRE as a function of number of iterations $(d=2)$.
  • Figure 2: Final RRE as a function of $a$ (noise level).
  • Figure 3: MNIST, $\lambda=10^{-2}$.

Theorems & Definitions (41)

  • Definition 1: Qi05
  • Remark 2
  • Remark 3
  • Lemma 4
  • proof
  • Proposition 5
  • Lemma 6
  • proof
  • Lemma 7
  • proof
  • ...and 31 more