Convex Relaxation for Solving Large-Margin Classifiers in Hyperbolic Space

TL;DR

This work tackles the challenge of large-margin learning in hyperbolic space by transforming the hyperbolic SVM problem into a polynomial QCQP and applying two convex relaxations: semidefinite programming and sparse moment-sum-of-squares. The proposed approaches yield tighter, more reliable separators than gradient-based methods across synthetic and real hyperbolic datasets, with Moment-SOS often delivering the smallest optimality gaps and best generalization in many settings. However, scalability remains a critical issue, motivating future work on hybrid algorithms, dual formulations, kernels, and scalable variants. Overall, the paper demonstrates that convex relaxation techniques can meaningfully certify and improve hyperbolic classification performance, particularly for hierarchical and graph-structured data embedded in \\mathbb{H}^d.

Abstract

Hyperbolic spaces have increasingly been recognized for their outstanding performance in handling data with inherent hierarchical structures compared to their Euclidean counterparts. However, learning in hyperbolic spaces poses significant challenges. In particular, extending support vector machines to hyperbolic spaces is in general a constrained non-convex optimization problem. Previous and popular attempts to solve hyperbolic SVMs, primarily using projected gradient descent, are generally sensitive to hyperparameters and initializations, often leading to suboptimal solutions. In this work, by first rewriting the problem into a polynomial optimization, we apply semidefinite relaxation and sparse moment-sum-of-squares relaxation to effectively approximate the optima. From extensive empirical experiments, these methods are shown to perform better than the projected gradient descent approach.

Figures (14)

• Figure 1: Straight line (red) on Lorentz manifold $\mathbb{H}^2$ as the intersection between a hyperplane and the manifold, presented similarly in cho2019large.
• Figure 2: Star-shaped Sparsity pattern in \ref{['eq:soft-margin-first-order-moment']} visualized with $n =4$
• Figure 3: Three Synthetic Gaussian (top row) and Three Tree Embeddings (bottom row). All features are in $\mathbb{H}^2$ but visualized through stereographic projection on $\mathbb{B}^2$. Different colors represent different classes. For tree dataset, the graph connections are also visualized but not used in training. The selected tree embeddings come directly from mishne2023numerical.
• Figure 4: Real datasets embedded on $\mathbb{H}^2$ visualized in $\mathbb{B}^2$. Different colors represent different classes. The first three (football, karate, polbooks) are graph embeddings; the latter two (cifar10, fashion mnist) on the top row are standard ML benchmarks; the last 5 dataset are single-cell sequencing data embedded on $\mathbb{H}^2$ for cell type discovery and miscellaneous biomedical usages.
• Figure 5: Decision boundary obtained by each method on one fold of train test split on Gaussian 1 dataset in \ref{['fig:synthetic-data']}. While SDP and moment overlap, they differ from the PGD solution.

Theorems & Definitions (3)

• Definition 1
• Remark 1
• Remark 2