Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks

Mingze Wang, Weinan E

TL;DR

This work conducts a systematic study of the expressive power of MoEs in modeling complex tasks with two common structural priors: low-dimensionality and sparsity, and proves that they can efficiently approximate functions supported on low-dimensional manifolds, overcoming the curse of dimensionality.

Abstract

Mixture-of-experts networks (MoEs) have demonstrated remarkable efficiency in modern deep learning. Despite their empirical success, the theoretical foundations underlying their ability to model complex tasks remain poorly understood. In this work, we conduct a systematic study of the expressive power of MoEs in modeling complex tasks with two common structural priors: low-dimensionality and sparsity. For shallow MoEs, we prove that they can efficiently approximate functions supported on low-dimensional manifolds, overcoming the curse of dimensionality. For deep MoEs, we show that $\mathcal{O}(L)$-layer MoEs with $E$ experts per layer can approximate piecewise functions comprising $E^L$ pieces with compositional sparsity, i.e., they can exhibit an exponential number of structured tasks. Our analysis reveals the roles of critical architectural components and hyperparameters in MoEs, including the gating mechanism, expert networks, the number of experts, and the number of layers, and offers natural suggestions for MoE variants.

On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks

TL;DR

This work conducts a systematic study of the expressive power of MoEs in modeling complex tasks with two common structural priors: low-dimensionality and sparsity, and proves that they can efficiently approximate functions supported on low-dimensional manifolds, overcoming the curse of dimensionality.

Abstract

Mixture-of-experts networks (MoEs) have demonstrated remarkable efficiency in modern deep learning. Despite their empirical success, the theoretical foundations underlying their ability to model complex tasks remain poorly understood. In this work, we conduct a systematic study of the expressive power of MoEs in modeling complex tasks with two common structural priors: low-dimensionality and sparsity. For shallow MoEs, we prove that they can efficiently approximate functions supported on low-dimensional manifolds, overcoming the curse of dimensionality. For deep MoEs, we show that -layer MoEs with experts per layer can approximate piecewise functions comprising pieces with compositional sparsity, i.e., they can exhibit an exponential number of structured tasks. Our analysis reveals the roles of critical architectural components and hyperparameters in MoEs, including the gating mechanism, expert networks, the number of experts, and the number of layers, and offers natural suggestions for MoE variants.

Paper Structure

This paper contains 28 sections, 7 theorems, 62 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Basic notations.
  5. MoE networks
  6. MoE components.
  7. MoE operation.
  8. Classical approximation results
  9. Theory for Shallow MoE Networks
  10. Manifold in Euclidean Space
  11. Theoretical results and insights
  12. Practical Suggestions
  13. Theory for Multi-layer MoE Networks
  14. Piecewise function with compositional sparsity
  15. Warm-up: Piecewise function on $E^L$ unit cubes with compositional sparsity.
  16. ...and 13 more sections

Key Result

Theorem 3.1

Let $D,K\in\mathbb{N}$, and $\Omega\subset\mathbb{R}^D$ be compact. For any $f\in\mathcal{C}^K(\Omega)$ and $m\in\mathbb{N}$, there exits a two-layer ReLU neural network $f_m$ with $m$ hidden neurons such that

Figures (3)

  • Figure 1: Illustration of an MoE layer.
  • Figure 2: A $d$-dimensional manifold $\mathcal{M}$ in $\mathbb{R}^D$.
  • Figure 3: Illustration of Eq. \ref{['example: piecewise function, 1-dim']}: a piecewise function $f$ with compositional sparsity on $3^2 = 9$ unit cubes. The function $f(\boldsymbol{x}) = f_{1,i_1}(x_1) f_{2,i_2}(x_2)$ is composed from 6 subfunctions: $f_{1,i}(z) = i(i - 1 - z)(z - i)$ and $f_{2,i}(z) = i(i - 1 - z)^2(z - i)^2$ on $z \in [i - 1, i]$ for $i \in [3]$. Although $f$ is smooth within each region, it is only $0$-order continuous on $[0,3] \times [0,3]$.

Theorems & Definitions (15)

  • Theorem 3.1
  • Definition 4.1: Chart and Atlas
  • Definition 4.2: Smooth manifold
  • Definition 4.3: Partition of unity
  • Theorem 4.4: Existence of a partition of unity
  • Definition 4.5: Function on the manifold
  • Example 4.6: Highly regular atlas
  • Definition 4.7: Regular atlas
  • Theorem 4.8: Main result
  • Corollary 4.9: Special case, highly regular atlas
  • ...and 5 more