Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Support-Set Algorithm for Optimization Problems with Nonnegative and Orthogonal Constraints

Lei Wang, Xin Liu, Xiaojun Chen

TL;DR

The paper tackles optimization over the nonnegative orthogonal set $\mathcal{O}^{n,p}_{+}$, where each row has at most one nonzero, posing a combinatorial challenge. It introduces a feasibility-preserving Support-Set algorithm that solves fixed-support proximal-linearized subproblems in closed form and uses a novel support-update mechanism to explore better sparsity patterns, while maintaining feasibility at every step. The authors prove global convergence to a first-order stationary point with iteration complexity $O(\epsilon^{-2})$ and establish finite support identification, and they demonstrate large practical gains on real tasks such as nonnegative PCA, clustering, and community detection. The approach exploits the problem structure to achieve substantial speedups over existing penalty-based and infeasible methods, offering a scalable framework for sparsity-constrained, orthogonality-preserving optimization in data science applications.

Abstract

In this paper, we investigate optimization problems with nonnegative and orthogonal constraints, where any feasible matrix of size $n \times p$ exhibits a sparsity pattern such that each row accommodates at most one nonzero entry. Our analysis demonstrates that, by fixing the support set, the global solution of the minimization subproblem for the proximal linearization of the objective function can be computed in closed form with at most $n$ nonzero entries. Exploiting this structural property offers a powerful avenue for dramatically enhancing computational efficiency. Guided by this insight, we propose a support-set algorithm preserving strictly the feasibility of iterates. A central ingredient is a strategically devised update scheme for support sets that adjusts the placement of nonzero entries. We establish the global convergence of the support-set algorithm to a first-order stationary point, and show that its iteration complexity required to reach an $ε$-approximate first-order stationary point is $O (ε^{-2})$. Numerical results are strongly in favor of our algorithm in real-world applications, including nonnegative PCA, clustering, and community detection.

A Support-Set Algorithm for Optimization Problems with Nonnegative and Orthogonal Constraints

TL;DR

The paper tackles optimization over the nonnegative orthogonal set , where each row has at most one nonzero, posing a combinatorial challenge. It introduces a feasibility-preserving Support-Set algorithm that solves fixed-support proximal-linearized subproblems in closed form and uses a novel support-update mechanism to explore better sparsity patterns, while maintaining feasibility at every step. The authors prove global convergence to a first-order stationary point with iteration complexity and establish finite support identification, and they demonstrate large practical gains on real tasks such as nonnegative PCA, clustering, and community detection. The approach exploits the problem structure to achieve substantial speedups over existing penalty-based and infeasible methods, offering a scalable framework for sparsity-constrained, orthogonality-preserving optimization in data science applications.

Abstract

In this paper, we investigate optimization problems with nonnegative and orthogonal constraints, where any feasible matrix of size exhibits a sparsity pattern such that each row accommodates at most one nonzero entry. Our analysis demonstrates that, by fixing the support set, the global solution of the minimization subproblem for the proximal linearization of the objective function can be computed in closed form with at most nonzero entries. Exploiting this structural property offers a powerful avenue for dramatically enhancing computational efficiency. Guided by this insight, we propose a support-set algorithm preserving strictly the feasibility of iterates. A central ingredient is a strategically devised update scheme for support sets that adjusts the placement of nonzero entries. We establish the global convergence of the support-set algorithm to a first-order stationary point, and show that its iteration complexity required to reach an -approximate first-order stationary point is . Numerical results are strongly in favor of our algorithm in real-world applications, including nonnegative PCA, clustering, and community detection.

Paper Structure

This paper contains 23 sections, 8 theorems, 84 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Prior and Related Works
  3. Contribution
  4. Organization
  5. Preliminaries
  6. Basic Notations
  7. Stationarity Condition
  8. Problem \ref{['opt:stplus']} With a Fixed Support Set
  9. Algorithm Development
  10. Refined Strategy for Zero Rows
  11. Update Scheme for Support Sets
  12. Complete Framework
  13. Convergence Analysis
  14. Auxiliary Results
  15. Global Convergence
  16. ...and 8 more sections

Key Result

Lemma 3.1

Suppose that $Z \in \mathcal{O}^{n,p}_{+}$ and $X \in \mathcal{O}^{n,p}_{+}$ satisfy $\bar{f}_{Z} (X) \leq \bar{f}_{Z} (Z)$. Then we have

Figures (3)

  • Figure 1: Numerical comparison of different algorithms for solving nonnegative PCA problems.
  • Figure 2: Numerical comparison of different algorithms for clustering.
  • Figure 3: Numerical comparison of different algorithms for community detection.

Theorems & Definitions (19)

  • Definition 2.1
  • Definition 2.2
  • Lemma 3.1
  • proof
  • Proposition 3.2
  • Remark 1
  • proof
  • Lemma 5.1
  • proof
  • Corollary 5.2
  • ...and 9 more