Hyperbolic Random Forests

TL;DR

This work proposes to generalize the well-known random forests to hyperbolic space by redefining the notion of a split using horospheres and outlines a new method for combining classes based on their lowest common ancestor and a class-balanced version of the large-margin loss.

Abstract

Hyperbolic space is becoming a popular choice for representing data due to the hierarchical structure - whether implicit or explicit - of many real-world datasets. Along with it comes a need for algorithms capable of solving fundamental tasks, such as classification, in hyperbolic space. Recently, multiple papers have investigated hyperbolic alternatives to hyperplane-based classifiers, such as logistic regression and SVMs. While effective, these approaches struggle with more complex hierarchical data. We, therefore, propose to generalize the well-known random forests to hyperbolic space. We do this by redefining the notion of a split using horospheres. Since finding the globally optimal split is computationally intractable, we find candidate horospheres through a large-margin classifier. To make hyperbolic random forests work on multi-class data and imbalanced experiments, we furthermore outline a new method for combining classes based on their lowest common ancestor and a class-balanced version of the large-margin loss. Experiments on standard and new benchmarks show that our approach outperforms both conventional random forest algorithms and recent hyperbolic classifiers.

Figures (6)

• Figure 1: Motivation for hyperbolic random forests. Two splits of (a) an axis-aligned Euclidean, (b) an oblique Euclidean, and (b) a hyperbolic decision tree of depth two. The data is a continuous embedding of a tree split into three nested classes. The inductive biases of linear decision boundaries are inappropriate for efficiently capturing the underlying geometry, and more splits are needed to be effective.
• Figure 2: Information gain from hierarchical splits. The best (a) one-versus-rest split and (b) split found with our hyperclass heuristic. We find splits with a higher information gain by considering splits where more than one class is seen as positive.
• Figure 3: Importance of class balancing. The optimal solution on a binary classification problem found (left) without and (right) with class-balancing. The left horosphere has a large negative $\mathbf{b}$ and does not split the data. In contrast, the right horosphere has a higher loss but a positive information gain. Class labels are given by marker shape. The points and color bar are colored by loss; the background is colored by distance to the horosphere.
• Figure 4: Visualizing HoroRF & OblRF splits. We show five splits for HoroRF, a basic version of HoroRF without class-balancing and hyperclasses, OblRF on the original hyperbolic embeddings, and OblRF after mapping the data to Euclidean space. The split with the highest impurity is chosen at every level to visualize the next level. Splitting in hyperbolic space with balanced and multi-level horospheres obtains good splits for hyperbolic classification.
• Figure 5: Analyzing ways to reduce the computational time of HoroRF on polbooks. In (a), we show that the optimal performance is already reached with around 20 trees. In (b), we show that by reducing the number of optimization iterations of HoroSVM, we can further reduce computational time while maintaining high performance.