Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GeoMM: On Geodesic Perspective for Multi-modal Learning

Shibin Mei, Hang Wang, Bingbing Ni

TL;DR

GeoMM introduces geodesic distance as a novel distance metric for multimodal learning, arguing that complex data manifolds are not well captured by traditional Euclidean or spherical distances. The method constructs a local-neighborhood graph over a large sample pool, computes geodesic distances as shortest-path distances, and employs a hierarchical graph to scale computation with incremental updates. The approach is integrated into multimodal pretraining with angle-normalized geodesic loss, replacing cosine distance in the ITC objective and yielding improved performance across image-text retrieval, VQA, NLVR2, visual grounding, and related tasks. Theoretical analysis provides sparsity and connectivity guarantees for the hierarchical graph, and extensive experiments demonstrate strong generalization and efficiency gains over strong baselines, validating the practical value of the geodesic perspective in multimodal learning.

Abstract

Geodesic distance serves as a reliable means of measuring distance in nonlinear spaces, and such nonlinear manifolds are prevalent in the current multimodal learning. In these scenarios, some samples may exhibit high similarity, yet they convey different semantics, making traditional distance metrics inadequate for distinguishing between positive and negative samples. This paper introduces geodesic distance as a novel distance metric in multi-modal learning for the first time, to mine correlations between samples, aiming to address the limitations of common distance metric. Our approach incorporates a comprehensive series of strategies to adapt geodesic distance for the current multimodal learning. Specifically, we construct a graph structure to represent the adjacency relationships among samples by thresholding distances between them and then apply the shortest-path algorithm to obtain geodesic distance within this graph. To facilitate efficient computation, we further propose a hierarchical graph structure through clustering and combined with incremental update strategies for dynamic status updates. Extensive experiments across various downstream tasks validate the effectiveness of our proposed method, demonstrating its capability to capture complex relationships between samples and improve the performance of multimodal learning models.

GeoMM: On Geodesic Perspective for Multi-modal Learning

TL;DR

GeoMM introduces geodesic distance as a novel distance metric for multimodal learning, arguing that complex data manifolds are not well captured by traditional Euclidean or spherical distances. The method constructs a local-neighborhood graph over a large sample pool, computes geodesic distances as shortest-path distances, and employs a hierarchical graph to scale computation with incremental updates. The approach is integrated into multimodal pretraining with angle-normalized geodesic loss, replacing cosine distance in the ITC objective and yielding improved performance across image-text retrieval, VQA, NLVR2, visual grounding, and related tasks. Theoretical analysis provides sparsity and connectivity guarantees for the hierarchical graph, and extensive experiments demonstrate strong generalization and efficiency gains over strong baselines, validating the practical value of the geodesic perspective in multimodal learning.

Abstract

Geodesic distance serves as a reliable means of measuring distance in nonlinear spaces, and such nonlinear manifolds are prevalent in the current multimodal learning. In these scenarios, some samples may exhibit high similarity, yet they convey different semantics, making traditional distance metrics inadequate for distinguishing between positive and negative samples. This paper introduces geodesic distance as a novel distance metric in multi-modal learning for the first time, to mine correlations between samples, aiming to address the limitations of common distance metric. Our approach incorporates a comprehensive series of strategies to adapt geodesic distance for the current multimodal learning. Specifically, we construct a graph structure to represent the adjacency relationships among samples by thresholding distances between them and then apply the shortest-path algorithm to obtain geodesic distance within this graph. To facilitate efficient computation, we further propose a hierarchical graph structure through clustering and combined with incremental update strategies for dynamic status updates. Extensive experiments across various downstream tasks validate the effectiveness of our proposed method, demonstrating its capability to capture complex relationships between samples and improve the performance of multimodal learning models.
Paper Structure (25 sections, 2 theorems, 23 equations, 6 figures, 6 tables, 2 algorithms)

This paper contains 25 sections, 2 theorems, 23 equations, 6 figures, 6 tables, 2 algorithms.

Table of Contents

  1. 1 Introduction
  2. Related Work
  3. Multi-modal Learning
  4. Geodesic Distance
  5. Methodology
  6. Preliminary
  7. Geodesic View
  8. Hierarchical Structure
  9. Algorithm Adaptation
  10. Implementation Details
  11. Theoretical Analysis
  12. Experiments
  13. Downstream Tasks
  14. Image and Text Retrieval
  15. VQA, NLVR2, VE and Grounding
  16. ...and 10 more sections

Key Result

Theorem 1

(The number of connected components) For a layer of the hierarchical graph containing $N$ cluster centers, with each point having $\sigma$ neighbors, the number of connected components $\mathcal{E}$ in this layer satisfies:

Figures (6)

  • Figure 1: The motivation of leveraging geodesic metric. Based on the topological structures, geodesic distance can depict sample similarity more accurately.
  • Figure 2: An overview of our proposed hierarchical graph structure, where we display the clusters and geodesic paths.
  • Figure 3: Visual grounding visualization (left) and performance of weakly-supervised visual grounding on RefCOCO+ (right).
  • Figure 4: Generalization on other multimodal methods. We report R@1 accuracy of text retrieval (TR) and image retrieval (IR) on COCO with zero-shot setting.
  • Figure 5: Generalization on other self-supervised methods. We report the Top1 accuracy of linear classification on ImageNet dataset. (ResNet50 crop 224$\times$224).
  • ...and 1 more figures

Theorems & Definitions (4)

  • Definition 1
  • Definition 2
  • Theorem 1
  • Theorem 2