Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Riemannian Bilevel Optimization with Gradient Aggregation

Zhuo Chen, Xinjian Xu, Shihui Ying, Tieyong Zeng

TL;DR

This work addresses RBLO with structural constraints by mapping variables to Riemannian manifolds and introducing BDA to jointly coordinate upper- and lower-level updates. A Lagrangian/KKT-based hypergradient on manifolds is derived, and a unified update uses retractions on Grassmann-type manifolds with Barzilai–Borwein step sizes, switching to diminishing steps to guarantee convergence. The authors prove convergence under geodesic convexity and Lipschitz smoothness and demonstrate superior performance on multi-view hypergraph spectral clustering (MVHSC) using the 3sources dataset, outperforming Euclidean and existing manifold baselines. The approach offers a principled, structure-preserving optimization framework with practical gains in hierarchical decision-making tasks where constraints induce manifold geometry.

Abstract

Bilevel optimization (BLO) offers a principled framework for hierarchical decision-making and has been widely applied in machine learning tasks such as hyperparameter optimization and meta-learning. While existing BLO methods are mostly developed in Euclidean spaces, many real-world problems involve structural constraints. In this paper, we propose a Riemannian bilevel optimization (RBLO) algorithm that incorporates a bilevel descent aggregation (BDA) scheme to jointly coordinate upper- and lower-level updates. Concretely, first we abstract the constraints in the BLO to a manifold structure and then transform the constrained BLO be a unconstrained RBLO problem. Second, to address limitations of existing RBLO methods, particularly the restrictive assumptions required for convergence, we reformulate the bilevel problem using smooth manifold mappings and provide a convergence analysis under the conditions of geodesic convexity and Lipschitz smoothness. Finally, we recall the multi-view hypergraph spectral clustering task, and evaluate the proposed approach on 3sources data sets. The numerical results validate the superior performance over Euclidean and manifold-based baselines.

Riemannian Bilevel Optimization with Gradient Aggregation

TL;DR

This work addresses RBLO with structural constraints by mapping variables to Riemannian manifolds and introducing BDA to jointly coordinate upper- and lower-level updates. A Lagrangian/KKT-based hypergradient on manifolds is derived, and a unified update uses retractions on Grassmann-type manifolds with Barzilai–Borwein step sizes, switching to diminishing steps to guarantee convergence. The authors prove convergence under geodesic convexity and Lipschitz smoothness and demonstrate superior performance on multi-view hypergraph spectral clustering (MVHSC) using the 3sources dataset, outperforming Euclidean and existing manifold baselines. The approach offers a principled, structure-preserving optimization framework with practical gains in hierarchical decision-making tasks where constraints induce manifold geometry.

Abstract

Bilevel optimization (BLO) offers a principled framework for hierarchical decision-making and has been widely applied in machine learning tasks such as hyperparameter optimization and meta-learning. While existing BLO methods are mostly developed in Euclidean spaces, many real-world problems involve structural constraints. In this paper, we propose a Riemannian bilevel optimization (RBLO) algorithm that incorporates a bilevel descent aggregation (BDA) scheme to jointly coordinate upper- and lower-level updates. Concretely, first we abstract the constraints in the BLO to a manifold structure and then transform the constrained BLO be a unconstrained RBLO problem. Second, to address limitations of existing RBLO methods, particularly the restrictive assumptions required for convergence, we reformulate the bilevel problem using smooth manifold mappings and provide a convergence analysis under the conditions of geodesic convexity and Lipschitz smoothness. Finally, we recall the multi-view hypergraph spectral clustering task, and evaluate the proposed approach on 3sources data sets. The numerical results validate the superior performance over Euclidean and manifold-based baselines.
Paper Structure (12 sections, 8 theorems, 51 equations, 2 figures, 1 table, 1 algorithm)

This paper contains 12 sections, 8 theorems, 51 equations, 2 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Contributions and Organization
  4. Preliminaries
  5. A Riemannian Bilevel Aggregation Algorithm
  6. Hypergradient
  7. Step Size and Retraction
  8. Convergence Analysis
  9. Numerical Experiment
  10. Problem Statement and Dataset
  11. Numerical Result
  12. Conclusions

Key Result

theorem 1

Suppose both the above UL and LL objective functions convergence properties hold and $f(\mathbf{x},\mathbf{y})$ is continuous on $\mathcal{M}\times\mathcal{N}$. Let $\mathbf{x}_K$ be a $\epsilon_K-$minimun of $\varphi_K(\mathbf{x})$, i.e., $\varphi_K(\mathbf{x}_K)\le\varphi_K(\mathbf{x})+\epsilon_K,

Figures (2)

  • Figure 1: Comparison of the LL objective value over iterations during the 1st and 2nd UL iterations for four algorithms.
  • Figure 2: Distributions of UL objective values across algorithms and different $(K_1, K_2)$ configurations.

Theorems & Definitions (22)

  • definition thmcounterdefinition
  • definition thmcounterdefinition
  • definition thmcounterdefinition
  • definition thmcounterdefinition
  • definition thmcounterdefinition
  • definition thmcounterdefinition
  • theorem 1
  • proof
  • theorem 2
  • proof
  • ...and 12 more