Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust low-rank estimation with multiple binary responses using pairwise AUC loss

The Tien Mai

TL;DR

This work tackles learning with many binary outcomes by leveraging a low-rank coefficient matrix to capture shared structure across tasks and by directly optimizing a pairwise AUC surrogate to improve discrimination under class imbalance. It introduces a projected gradient descent algorithm with truncated SVD to solve the rank-constrained, AUC-based objective, and provides nonasymptotic convergence guarantees along with a minimax-optimal statistical error bound. Theoretical results rely on Restricted Strong Convexity and Restricted Smoothness on low-rank subspaces, ensuring linear convergence to a noise-dominated neighborhood and a $O\left(\sqrt{\frac{r(p+q)}{n}}\right)$ estimation rate. Empirically, the robust AUC-based reduced-rank regression matches standard methods under clean data but shows substantial improvements in estimation accuracy and ranking performance under label noise and covariate contamination, demonstrating robustness and scalability in high-dimensional, multi-output settings.

Abstract

Multiple binary responses arise in many modern data-analytic problems. Although fitting separate logistic regressions for each response is computationally attractive, it ignores shared structure and can be statistically inefficient, especially in high-dimensional and class-imbalanced regimes. Low-rank models offer a natural way to encode latent dependence across tasks, but existing methods for binary data are largely likelihood-based and focus on pointwise classification rather than ranking performance. In this work, we propose a unified framework for learning with multiple binary responses that directly targets discrimination by minimizing a surrogate loss for the area under the ROC curve (AUC). The method aggregates pairwise AUC surrogate losses across responses while imposing a low-rank constraint on the coefficient matrix to exploit shared structure. We develop a scalable projected gradient descent algorithm based on truncated singular value decomposition. Exploiting the fact that the pairwise loss depends only on differences of linear predictors, we simplify computation and analysis. We establish non-asymptotic convergence guarantees, showing that under suitable regularity conditions, leading to linear convergence up to the minimax-optimal statistical precision. Extensive simulation studies demonstrate that the proposed method is robust in challenging settings such as label switching and data contamination and consistently outperforms likelihood-based approaches.

Robust low-rank estimation with multiple binary responses using pairwise AUC loss

TL;DR

This work tackles learning with many binary outcomes by leveraging a low-rank coefficient matrix to capture shared structure across tasks and by directly optimizing a pairwise AUC surrogate to improve discrimination under class imbalance. It introduces a projected gradient descent algorithm with truncated SVD to solve the rank-constrained, AUC-based objective, and provides nonasymptotic convergence guarantees along with a minimax-optimal statistical error bound. Theoretical results rely on Restricted Strong Convexity and Restricted Smoothness on low-rank subspaces, ensuring linear convergence to a noise-dominated neighborhood and a estimation rate. Empirically, the robust AUC-based reduced-rank regression matches standard methods under clean data but shows substantial improvements in estimation accuracy and ranking performance under label noise and covariate contamination, demonstrating robustness and scalability in high-dimensional, multi-output settings.

Abstract

Multiple binary responses arise in many modern data-analytic problems. Although fitting separate logistic regressions for each response is computationally attractive, it ignores shared structure and can be statistically inefficient, especially in high-dimensional and class-imbalanced regimes. Low-rank models offer a natural way to encode latent dependence across tasks, but existing methods for binary data are largely likelihood-based and focus on pointwise classification rather than ranking performance. In this work, we propose a unified framework for learning with multiple binary responses that directly targets discrimination by minimizing a surrogate loss for the area under the ROC curve (AUC). The method aggregates pairwise AUC surrogate losses across responses while imposing a low-rank constraint on the coefficient matrix to exploit shared structure. We develop a scalable projected gradient descent algorithm based on truncated singular value decomposition. Exploiting the fact that the pairwise loss depends only on differences of linear predictors, we simplify computation and analysis. We establish non-asymptotic convergence guarantees, showing that under suitable regularity conditions, leading to linear convergence up to the minimax-optimal statistical precision. Extensive simulation studies demonstrate that the proposed method is robust in challenging settings such as label switching and data contamination and consistently outperforms likelihood-based approaches.
Paper Structure (16 sections, 8 theorems, 52 equations, 4 tables)

This paper contains 16 sections, 8 theorems, 52 equations, 4 tables.

Table of Contents

  1. Introduction
  2. Model and method
  3. Model
  4. Proposed method
  5. Projected gradient descent algorithm
  6. Theoretical analysis
  7. Assumptions
  8. Convergence of the algorithm
  9. Statistical analysis
  10. Simulation examples
  11. Simulation setting
  12. Simulation results
  13. Closing discussion
  14. Proofs
  15. Proof of Theorem 1
  16. ...and 1 more sections

Key Result

Theorem 1

Suppose Assumptions ass:RSC--ass:RSM hold and let $\eta = 1/L$. Let $\{B^{(t)}\}_{t\ge0}$ be the iterates defined by eq:pgd_update, and assume $\mathrm{rank}(B^\star)\le r$. Then there exist constants $\rho \in (0,1)$ and $C>0$ such that In particular, the iterates converge linearly to a neighborhood of $B^\star$ whose radius is proportional to the statistical error $\|\nabla \widehat{\mathcal{L}

Theorems & Definitions (16)

  • Theorem 1: Linear Convergence of PGD
  • Theorem 2: Error Bound
  • Lemma 1: Low-rank projection error
  • proof : Proof of Lemma \ref{['lem:projection']}
  • Lemma 2: Gradient Step Contraction
  • proof : Proof of Lemma \ref{['lem:gradient_contraction']}
  • proof : Proof of Theorem \ref{['thm:pgd_convergence']}
  • proof : Proof of Theorem \ref{['theorem_statistical_error']}
  • Lemma 3: Uniform Gradient Concentration
  • proof : Proof of Lemma \ref{['lemma:concentration']}
  • ...and 6 more