Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Experts

Buze Zhang, Jinkai Tao, Zilang Zeng, Neil He, Ali Maatouk, Menglin Yang, Rex Ying

TL;DR

Mixture of Space (MoS) is proposed, a unified framework that leverages multiple geometric spaces simultaneously to learn richer, curvature-aware representations, and MoSLoRA is developed, which extends Low-Rank Adaptation with heterogeneous geometric experts, enabling models to dynamically select or combine appropriate geometric spaces based on input context.

Abstract

Large Language Models (LLMs) have achieved remarkable progress, with Parameter-Efficient Fine-Tuning (PEFT) emerging as a key technique for downstream task adaptation. However, existing PEFT methods mainly operate in Euclidean space, fundamentally limiting their capacity to capture complex geometric structures inherent in language data. While alternative geometric spaces, like hyperbolic geometries for hierarchical data and spherical manifolds for circular patterns, offer theoretical advantages, forcing representations into a single manifold type ultimately limits expressiveness, even when curvature parameters are learnable. To address this, we propose Mixture of Space (MoS), a unified framework that leverages multiple geometric spaces simultaneously to learn richer, curvature-aware representations. Building on this scheme, we develop MoSLoRA, which extends Low-Rank Adaptation (LoRA) with heterogeneous geometric experts, enabling models to dynamically select or combine appropriate geometric spaces based on input context. Furthermore, to address the computational overhead of frequent manifold switching, we develop a lightweight routing mechanism. Moreover, we provide empirical insights into how curvature optimization impacts training stability and model performance. Our experiments across diverse benchmarks demonstrate that MoSLoRA consistently outperforms strong baselines, achieving up to 5.6% improvement on MATH500 and 15.9% on MAWPS.

Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Experts

TL;DR

Mixture of Space (MoS) is proposed, a unified framework that leverages multiple geometric spaces simultaneously to learn richer, curvature-aware representations, and MoSLoRA is developed, which extends Low-Rank Adaptation with heterogeneous geometric experts, enabling models to dynamically select or combine appropriate geometric spaces based on input context.

Abstract

Large Language Models (LLMs) have achieved remarkable progress, with Parameter-Efficient Fine-Tuning (PEFT) emerging as a key technique for downstream task adaptation. However, existing PEFT methods mainly operate in Euclidean space, fundamentally limiting their capacity to capture complex geometric structures inherent in language data. While alternative geometric spaces, like hyperbolic geometries for hierarchical data and spherical manifolds for circular patterns, offer theoretical advantages, forcing representations into a single manifold type ultimately limits expressiveness, even when curvature parameters are learnable. To address this, we propose Mixture of Space (MoS), a unified framework that leverages multiple geometric spaces simultaneously to learn richer, curvature-aware representations. Building on this scheme, we develop MoSLoRA, which extends Low-Rank Adaptation (LoRA) with heterogeneous geometric experts, enabling models to dynamically select or combine appropriate geometric spaces based on input context. Furthermore, to address the computational overhead of frequent manifold switching, we develop a lightweight routing mechanism. Moreover, we provide empirical insights into how curvature optimization impacts training stability and model performance. Our experiments across diverse benchmarks demonstrate that MoSLoRA consistently outperforms strong baselines, achieving up to 5.6% improvement on MATH500 and 15.9% on MAWPS.
Paper Structure (31 sections, 1 theorem, 32 equations, 4 figures, 10 tables)

This paper contains 31 sections, 1 theorem, 32 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Mixture of LoRA Experts
  4. Non-Euclidean and Curvature-Aware Modeling
  5. Preliminary
  6. Mixture of LoRA Experts
  7. Lorentz Model of Hyperbolic Space
  8. Tangent Space and Maps.
  9. Mixture of Space Experts Framework
  10. Mixture of Space Scheme
  11. Unified Stereographic Projection
  12. Mixture of Space Parameters-Efficient Tuning
  13. Experiments
  14. Experiment Setting
  15. Baselines Comparison
  16. ...and 16 more sections

Key Result

Lemma 4.1

Let $\kappa \neq 0$ and let $c>0$ be a variable scaling constant. Consider the unified projection and MoS transformation as $F_{\kappa}(\cdot)$ defined in Eq.unified_scheme, invproj, and proj. Applying a scaling factor $\gamma$ to the input before the inverse projection and rescaling the output by t

Figures (4)

  • Figure 1: The MoSLoRA architecture contains heterogeneous geometric experts unified in our MoS scheme, with various curvatures. Three geometric expert groups are embedded into the FFN layer and labeled in different colored blocks. The grouped auxiliary balancing enforces balanced routing within each space group while allowing free inter-space transitions, and remains fully reversible and differentiable.
  • Figure 2: Curvature dynamics of each geometric expert of different layers during training.
  • Figure 3: Token norm distribution and token frequency distribution of math reasoning and commonsenseQA datasets.
  • Figure 4: Efficiency comparison between exp/log and our MoS.

Theorems & Definitions (1)

  • Lemma 4.1: Scale-Invariant Equivalence of the Unified Projection