Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Riemannian Optimization on Relaxed Indicator Matrix Manifold

Jinghui Yuan, Fangyuan Xie, Feiping Nie, Xuelong Li

TL;DR

The paper addresses the NP-hard problem of optimizing indicator matrices by introducing a flexible relaxation that yields the Relaxed Indicator Matrix Manifold (RIM). It develops a complete Riemannian optimization toolbox on the RIM manifold, including three Retraction strategies and efficient gradient/Hessian mappings, achieving a theoretical and practical reduction from $O(n^3)$ to $O(n)$-type complexity relative to prior double stochastic methods. The framework is applied to graph cuts, notably Ratio Cut, with rigorous convergence analyses and state-of-the-art clustering performance demonstrated on large-scale datasets and image-denoising tasks. The work offers open-source code and shows substantial practical impact for scalable, geometry-respecting optimization in clustering and graph-based learning.

Abstract

The indicator matrix plays an important role in machine learning, but optimizing it is an NP-hard problem. We propose a new relaxation of the indicator matrix and prove that this relaxation forms a manifold, which we call the Relaxed Indicator Matrix Manifold (RIM manifold). Based on Riemannian geometry, we develop a Riemannian toolbox for optimization on the RIM manifold. Specifically, we provide several methods of Retraction, including a fast Retraction method to obtain geodesics. We point out that the RIM manifold is a generalization of the double stochastic manifold, and it is much faster than existing methods on the double stochastic manifold, which has a complexity of \( \mathcal{O}(n^3) \), while RIM manifold optimization is \( \mathcal{O}(n) \) and often yields better results. We conducted extensive experiments, including image denoising, with millions of variables to support our conclusion, and applied the RIM manifold to Ratio Cut, we provide a rigorous convergence proof and achieve clustering results that outperform the state-of-the-art methods. Our Code in \href{https://github.com/Yuan-Jinghui/Riemannian-Optimization-on-Relaxed-Indicator-Matrix-Manifold}{here}.

Riemannian Optimization on Relaxed Indicator Matrix Manifold

TL;DR

The paper addresses the NP-hard problem of optimizing indicator matrices by introducing a flexible relaxation that yields the Relaxed Indicator Matrix Manifold (RIM). It develops a complete Riemannian optimization toolbox on the RIM manifold, including three Retraction strategies and efficient gradient/Hessian mappings, achieving a theoretical and practical reduction from to -type complexity relative to prior double stochastic methods. The framework is applied to graph cuts, notably Ratio Cut, with rigorous convergence analyses and state-of-the-art clustering performance demonstrated on large-scale datasets and image-denoising tasks. The work offers open-source code and shows substantial practical impact for scalable, geometry-respecting optimization in clustering and graph-based learning.

Abstract

The indicator matrix plays an important role in machine learning, but optimizing it is an NP-hard problem. We propose a new relaxation of the indicator matrix and prove that this relaxation forms a manifold, which we call the Relaxed Indicator Matrix Manifold (RIM manifold). Based on Riemannian geometry, we develop a Riemannian toolbox for optimization on the RIM manifold. Specifically, we provide several methods of Retraction, including a fast Retraction method to obtain geodesics. We point out that the RIM manifold is a generalization of the double stochastic manifold, and it is much faster than existing methods on the double stochastic manifold, which has a complexity of \( \mathcal{O}(n^3) \), while RIM manifold optimization is \( \mathcal{O}(n) \) and often yields better results. We conducted extensive experiments, including image denoising, with millions of variables to support our conclusion, and applied the RIM manifold to Ratio Cut, we provide a rigorous convergence proof and achieve clustering results that outperform the state-of-the-art methods. Our Code in \href{https://github.com/Yuan-Jinghui/Riemannian-Optimization-on-Relaxed-Indicator-Matrix-Manifold}{here}.

Paper Structure

This paper contains 66 sections, 9 theorems, 170 equations, 8 figures, 11 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Riemannian Toolbox
  4. Definition of the Relaxed Indicator Matrix Manifold
  5. Riemannian Optimization Toolbox for the RIM Manifold
  6. Comparison Analysis of Time Complexity
  7. RIM Manifold for Graph Cut
  8. Experiments
  9. Experimental Setups
  10. Experiment 1 Setup
  11. Experiment 2 Setup
  12. Experiment 3 Setup
  13. Experiment 4 Setup
  14. Experimental Results
  15. Result of Experimental 1
  16. ...and 51 more sections

Key Result

Theorem 1

Our relaxed indicator matrix set $\mathcal{M} = \{ X \mid X 1_c = 1_n, l < X^T 1_n < u, X > 0 \}$ forms an embedded submanifold of the Euclidean space, with $\dim \mathcal{M} = (n-1)c$. We refer to it as the Relaxed Indicator Matrix Manifold. Proof in proof1

Figures (8)

  • Figure 1: Image Denoising Results, Noise Coefficient 0.3, $\xi = 0.3$.
  • Figure 2: Comparison of Loss Decrease for Optimization Algorithms on Real Datasets.
  • Figure 3: Comparison of running time for different Retraction algorithms.
  • Figure 4: mage Denoising Results.
  • Figure 5: Comparison of Loss Decrease for Optimization Algorithms on Real Datasets ($l=u$).
  • ...and 3 more figures

Theorems & Definitions (20)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Theorem 5
  • Theorem 6
  • Theorem 7
  • Theorem 8
  • Theorem 9
  • proof
  • ...and 10 more