A new perspective on low-rank optimization

TL;DR

This work tackles the challenge of obtaining strong lower bounds for low-rank optimization by introducing the Matrix Perspective Reformulation Technique (MPRT), a matrix-analytic generalization of the classical perspective reformulation used in mixed-integer optimization. By defining a matrix perspective function $g_f$ and coupling it with orthogonal projection matrices to encode rank, the authors characterize convex hulls of low-rank sets and derive semidefinite relaxations that are often tight and scalable. They provide a broad set of examples (spectral constraints, quadratic penalties, entropy-based penalties, and CP-relaxations) and demonstrate concrete relaxations for matrix completion, tensor completion, low-rank factor analysis, and D-optimal experimental design. The numerical results on reduced rank regression, non-negative matrix factorization, and experimental design illustrate improved bound quality and scalability, validating MPRT as a practical tool for obtaining tractable, high-quality lower bounds in diverse low-rank problems.

Abstract

A key question in many low-rank problems throughout optimization, machine learning, and statistics is to characterize the convex hulls of simple low-rank sets and judiciously apply these convex hulls to obtain strong yet computationally tractable convex relaxations. We invoke the matrix perspective function - the matrix analog of the perspective function - and characterize explicitly the convex hull of epigraphs of simple matrix convex functions under low-rank constraints. Further, we combine the matrix perspective function with orthogonal projection matrices-the matrix analog of binary variables which capture the row-space of a matrix-to develop a matrix perspective reformulation technique that reliably obtains strong relaxations for a variety of low-rank problems, including reduced rank regression, non-negative matrix factorization, and factor analysis. Moreover, we establish that these relaxations can be modeled via semidefinite constraints and thus optimized over tractably. The proposed approach parallels and generalizes the perspective reformulation technique in mixed-integer optimization and leads to new relaxations for a broad class of problems.

Figures (4)

• Figure 1: Comparative performance, as the number of samples $m$ increases, of formulations \ref{['eqn:rrr_persp']} (Persp, in blue), \ref{['eqn:rrr_dcl']} (DCL, in orange) and \ref{['eqn:rrr_nn']} (NN, in green), averaged over $100$ synthetic reduced rank regression instances where $n=p=50$, $k_{true}=10$. The hyperparameter $\mu$ was first cross-validated for all approaches separately.
• Figure 2: Average time to compute an optimal solution (left panel) and peak memory usage (right panel) vs. dimensionality $n=p$ for Problems \ref{['eqn:rrr_persp']} (Persp, in blue), \ref{['eqn:rrr_dcl']} (DCL. in orange) and \ref{['eqn:rrr_nn']} (NN, in green) over $20$ synthetic reduced rank regression instances where $k_{true}=10$.
• Figure 3: Average relative MSE and duality gap vs. target rank $k$ using the ALS heuristic (UB) and the MPRT relaxation (LB). Results are averaged over $100$ synthetic completely positive matrix factorization instances where $n=50$, $k_{true}=10$.
• Figure 4: Computational time to compute a feasible solution (ALS) and solve the relaxation (Semidefinite bound) vs. target rank $k$, averaged over $100$ synthetic completely positive matrix factorization instances where $n=50$, $k_{true}=10$.

Theorems & Definitions (46)

• lemma thmcounterlemma
• proof
• proposition thmcounterproposition
• remark thmcounterremark
• definition thmcounterdefinition
• remark thmcounterremark
• proposition thmcounterproposition
• proof
• proposition thmcounterproposition
• remark thmcounterremark