Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sparsity is Combinatorial Depth: Quantifying MoE Expressivity via Tropical Geometry

Ye Su, Huayi Tang, Zixuan Gong, Yong Liu

TL;DR

This work develops a tropical-geometry framework to quantify MoE expressivity, showing that Top-$k$ routing is algebraically isomorphic to the $k$-th elementary symmetric tropical polynomial and partitions inputs via the Normal Fan of a Hypersimplex, yielding a capacity scaling of $\Theta\left(\binom{N}{k}(kH)^{d_{in}}\right)$. It proves a Combinatorial Slicing Theorem with matching upper and lower bounds, highlighting a combinatorial depth that decouples capacity from inference cost. Under the Manifold Hypothesis, the authors define Effective Capacity and show a persistent combinatorial gain: $\Theta\left(\binom{N}{k}(kH)^{d_{eff}}\right)$, indicating MoE's resilience to geometric capacity collapse on low-dimensional data. Collectively, these results provide a rigorous geometric justification for the efficiency and expressivity advantages of sparse conditional computation in MoE architectures.

Abstract

While Mixture-of-Experts (MoE) architectures define the state-of-the-art, their theoretical success is often attributed to heuristic efficiency rather than geometric expressivity. In this work, we present the first analysis of MoE through the lens of tropical geometry, establishing that the Top-$k$ routing mechanism is algebraically isomorphic to the $k$-th elementary symmetric tropical polynomial. This isomorphism partitions the input space into the Normal Fan of a Hypersimplex, revealing that \textbf{sparsity is combinatorial depth} which scales geometric capacity by the binomial coefficient $\binom{N}{k}$. Moving beyond ambient bounds, we introduce the concept of \textit{Effective Capacity} under the Manifold Hypothesis. We prove that while dense networks suffer from capacity collapse on low-dimensional data, MoE architectures exhibit \textit{Combinatorial Resilience}, maintaining high expressivity via the transversality of routing cones. In this study, our framework unifies the discrete geometry of the Hypersimplex with the continuous geometry of neural functions, offering a rigorous theoretical justification for the topological supremacy of conditional computation.

Sparsity is Combinatorial Depth: Quantifying MoE Expressivity via Tropical Geometry

TL;DR

This work develops a tropical-geometry framework to quantify MoE expressivity, showing that Top- routing is algebraically isomorphic to the -th elementary symmetric tropical polynomial and partitions inputs via the Normal Fan of a Hypersimplex, yielding a capacity scaling of . It proves a Combinatorial Slicing Theorem with matching upper and lower bounds, highlighting a combinatorial depth that decouples capacity from inference cost. Under the Manifold Hypothesis, the authors define Effective Capacity and show a persistent combinatorial gain: , indicating MoE's resilience to geometric capacity collapse on low-dimensional data. Collectively, these results provide a rigorous geometric justification for the efficiency and expressivity advantages of sparse conditional computation in MoE architectures.

Abstract

While Mixture-of-Experts (MoE) architectures define the state-of-the-art, their theoretical success is often attributed to heuristic efficiency rather than geometric expressivity. In this work, we present the first analysis of MoE through the lens of tropical geometry, establishing that the Top- routing mechanism is algebraically isomorphic to the -th elementary symmetric tropical polynomial. This isomorphism partitions the input space into the Normal Fan of a Hypersimplex, revealing that \textbf{sparsity is combinatorial depth} which scales geometric capacity by the binomial coefficient . Moving beyond ambient bounds, we introduce the concept of \textit{Effective Capacity} under the Manifold Hypothesis. We prove that while dense networks suffer from capacity collapse on low-dimensional data, MoE architectures exhibit \textit{Combinatorial Resilience}, maintaining high expressivity via the transversality of routing cones. In this study, our framework unifies the discrete geometry of the Hypersimplex with the continuous geometry of neural functions, offering a rigorous theoretical justification for the topological supremacy of conditional computation.
Paper Structure (23 sections, 13 theorems, 90 equations, 2 figures, 1 table)

This paper contains 23 sections, 13 theorems, 90 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries: Neural Networks as Tropical Rational Functions
  3. The Tropical Semiring
  4. Neural Networks as Tropical Rational Functions
  5. Mixture-of-Experts: Formulation and Gating
  6. Sparse Top-$k$ Routing Mechanism.
  7. The Tropical Geometry of Routing
  8. Geometric Structure and Capacity Analysis
  9. The Geometry of Routing: From Hyperplanes to Hypersimplices
  10. Capacity Analysis: The Combinatorial Slicing Theorem
  11. Effective Capacity under the Manifold Hypothesis
  12. Conclusion
  13. Missing Proof of Theorem \ref{['thm:topk_geometry']}
  14. Missing Proof of Theorem \ref{['prop:dense_capacity']}
  15. Missing Proof of Theorem \ref{['thm:top1_capacity']}
  16. ...and 8 more sections

Key Result

Proposition 3.2

Let $\mathcal{N} = \{1, \dots, N\}$ be the set of expert indices, and let $z_i(\mathbf{x})$ denote the affine logit score for expert $i$. The decision boundaries of a Top-1 router form a Tropical Hypersurface, formally defined as the singular locus of the tropical polynomial $S_{\text{top1}}(\mathbf Geometrically, this locus constitutes the boundaries of a Voronoi Diagram (or Power Diagram), parti

Figures (2)

  • Figure 1: Schematic of a Top-$k$ Mixture-of-Experts Layer. The architecture separates routing logic from data processing. The router selects a sparse subset of experts (Active Set $\mathcal{I}_\mathbf{x}$), ensuring that inactive experts (grey, dashed) consume no computational resources. The final output is the weighted sum of the active experts.
  • Figure 2: Geometric Capacity via Duality and Activation Patterns. (Left) The Primal input space $\mathbb{R}^d$ is partitioned into convex polyhedral cells by a hyperplane arrangement $\mathcal{A} = \{H_1, H_2, H_3\}$, where each hyperplane corresponds to the decision boundary of a ReLU neuron. A specific linear region $\mathcal{R}_{\mathbf{v}}$ is defined by its activation pattern$\mathbf{s} \in \{0, 1\}^3$, representing the binary state of all neurons. For the shaded region $\mathcal{R}_{\mathbf{v}}$, the pattern is $\mathbf{s} = \{1, 1, 0\}$, indicating that neurons 1 and 2 are active ($H_1, H_2$ "on") while neuron 3 is inactive. (Right) In the Dual weight space, the layer is represented by a Newton Zonotope$\mathcal{Q}$, which is the Minkowski sum of line segments $\bigoplus_{i=1}^3 [\mathbf{0}, \mathbf{w}_i]$. The Duality Mapping$\mathcal{D}_\Phi$ (Legendre-Fenchel duality) associates primal linear regions $\mathcal{R}_{\mathbf{v}}$ with vertices $\mathbf{v}$ of the zonotope, calculated as the weighted sum of active neurons: $\mathbf{v} = \sum_{i} s_i \mathbf{w}_i$. For the highlighted region, $\mathcal{D}_\Phi(\mathcal{R}_{\mathbf{v}}) = \mathbf{w}_1 + \mathbf{w}_2$. While the exact counts differ between the affine arrangement and the dual zonotope, this framework demonstrates that the total geometric capacity is governed by the vertex complexity of the zonotope, sharing the same asymptotic growth rate $\Theta(H^d)$.

Theorems & Definitions (36)

  • Definition 2.1: Tropical Semiring
  • Definition 2.2: Tropical Polynomial
  • Definition 2.3: Tropical Rational Function
  • Definition 3.1: Hyperplane Arrangement
  • Proposition 3.2: Top-1 MoE Geometry: Tropical Hypersurfaces and Voronoi Diagrams
  • Remark 3.3: The Sparsity Gain
  • Theorem 3.4: Top-k MoE Geometry: Tropical Grassmannians and Order-$k$ Voronoi Diagrams
  • Remark 3.5: Geometric Intuition and Exactness
  • Definition 3.6: General Position
  • Definition 3.7: Zaslavsky's Function
  • ...and 26 more