HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Models

TL;DR

HyperET tackles the efficiency challenge in multi-modal large language models by shifting visual-language alignment to hyperbolic space, where the hyperbolic radius encodes granularity. It introduces learnable Möbius-based radius adjustments via parameter-efficient matrices (diagonal, block-diagonal, banded) that modulate pre-trained vision representations to match text at arbitrary granularity levels. Theoretical results (Theorems 1–2) establish precise radius control through Möbius operations, and extensive experiments show robust gains on fine-tuning and pre-training with under 1% additional parameters, including improvements on ScienceQA and TextVQA. The approach offers a scalable, interpretable route to granularity-aligned cross-modal learning with broad applicability to PEFT settings.

Abstract

Multi-modal large language models (MLLMs) have emerged as a transformative approach for aligning visual and textual understanding. They typically require extremely high computational resources (e.g., thousands of GPUs) for training to achieve cross-modal alignment at multi-granularity levels. We argue that a key source of this inefficiency lies in the vision encoders they widely equip with, e.g., CLIP and SAM, which lack the alignment with language at multi-granularity levels. To address this issue, in this paper, we leverage hyperbolic space, which inherently models hierarchical levels and thus provides a principled framework for bridging the granularity gap between visual and textual modalities at an arbitrary granularity level. Concretely, we propose an efficient training paradigm for MLLMs, dubbed as HyperET, which can optimize visual representations to align with their textual counterparts at an arbitrary granularity level through dynamic hyperbolic radius adjustment in hyperbolic space. HyperET employs learnable matrices with Möbius multiplication operations, implemented via three effective configurations: diagonal scaling matrices, block-diagonal matrices, and banded matrices, providing a flexible yet efficient parametrization strategy. Comprehensive experiments across multiple MLLM benchmarks demonstrate that HyperET consistently improves both existing pre-training and fine-tuning MLLMs clearly with less than 1\% additional parameters. Code is available at https://github.com/godlin-sjtu/HyperET

Figures (2)

• Figure 1: The schematic representation of HyperET. In HyperET, we efficiently train MLLMs in hyperbolic space by adjusting the hyperbolic radius using a tunable scaling matrix $\mathbf{W}_s$. Here, $\mathbf{W}_s$ can be configured into three parameter-efficient variants, i.e., Diagonal, Block-Diagonal and Banded.
• Figure 2: Visualization of hyperbolic radius changes in visual representation after training across different MLLM benchmarks. Normalizing the hyperbolic radius to a range of 0--1 facilitates comparison. A smaller hyperbolic radius corresponds to a more low granularity level of visual representation. "Zero-shot": maintaining the pre-trained weights of the vision encoder, i.e., CLIP, without additional training.