Enhance Hyperbolic Representation Learning via Second-order Pooling

TL;DR

Second-order pooling is introduced into hyperbolic representation learning, which enables the low-dimensional bilinear features to approximate the kernel function well in low-dimensional space and proposes a kernel approximation regularization.

Abstract

Hyperbolic representation learning is well known for its ability to capture hierarchical information. However, the distance between samples from different levels of hierarchical classes can be required large. We reveal that the hyperbolic discriminant objective forces the backbone to capture this hierarchical information, which may inevitably increase the Lipschitz constant of the backbone. This can hinder the full utilization of the backbone's generalization ability. To address this issue, we introduce second-order pooling into hyperbolic representation learning, as it naturally increases the distance between samples without compromising the generalization ability of the input features. In this way, the Lipschitz constant of the backbone does not necessarily need to be large. However, current off-the-shelf low-dimensional bilinear pooling methods cannot be directly employed in hyperbolic representation learning because they inevitably reduce the distance expansion capability. To solve this problem, we propose a kernel approximation regularization, which enables the low-dimensional bilinear features to approximate the kernel function well in low-dimensional space. Finally, we conduct extensive experiments on graph-structured datasets to demonstrate the effectiveness of the proposed method.

Figures (8)

• Figure 1: The hierarchical representation of data is retrieved using distances in representation learning.
• Figure 2: Demonstration of how hierarchical information is enhanced using bilinear pooling. Class 1 and Class 2 belong to the same subgroup, whereas Class 3 and Class 4 belong to another subgroup. On the left, the hierarchical information in the projection is not significant. On the right, after bilinear pooling, the hierarchy in the data representation is observed as clusters of the same subgroup become closer. The exact procedure to compute the inverse bilinear pooling is presented in Appendix A.
• Figure 3: Network architecture. An Euclidean GNN backbone is used for node feature extraction. For each node, bilinear features are expanded and further projected into hyperbolic space. The proposed kernel-based regularization is applied to obtain compact bilinear features.
• Figure 4: t-SNE visualization of the embeddings for (a) the Euclidean model and (b) the model incorporating the hyperbolic projection in the Cora dataset. The hierarchical structure of the dataset becomes visible after applying the hyperbolic projection. In (c), the embeddings are hierarchically clustered according to the class centroid distances obtained from (b).
• Figure 5: Impact of the curvature upon different levels of curvature in the accuracy for the citation datasets.