Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mean-field limit from general mixtures of experts to quantum neural networks

Anderson Melchor Hernandez, Davide Pastorello, Giacomo De Palma

TL;DR

Mean-field analysis of gradient-flow trained MoEs with identical experts is developed. The empirical distribution of expert parameters converges to $\mu_t$ solving a nonlinear continuity equation, with a convergence rate in Wasserstein distance: ${\mathcal{W}}_2^2$ bound $\mathbb{E}{\mathcal{W}}_2^2(\mu_{\Theta_t^N}, \mu_t) \le C (N^{-2/d} + N^{-1/2})$. The framework is instantiated for experts realized as parametric quantum circuits, where the Lipschitz constants yield $\alpha=\beta=1$ and the regime avoids lazy training, admitting representation learning. This work provides a rigorous mean-field bridge between finite-$N$ MoEs and a continuum limit, suggesting directions for time-uniform bounds and joint depth-width limits.

Abstract

In this work, we study the asymptotic behavior of Mixture of Experts (MoE) trained via gradient flow on supervised learning problems. Our main result establishes the propagation of chaos for a MoE as the number of experts diverges. We demonstrate that the corresponding empirical measure of their parameters is close to a probability measure that solves a nonlinear continuity equation, and we provide an explicit convergence rate that depends solely on the number of experts. We apply our results to a MoE generated by a quantum neural network.

Mean-field limit from general mixtures of experts to quantum neural networks

TL;DR

Mean-field analysis of gradient-flow trained MoEs with identical experts is developed. The empirical distribution of expert parameters converges to solving a nonlinear continuity equation, with a convergence rate in Wasserstein distance: bound . The framework is instantiated for experts realized as parametric quantum circuits, where the Lipschitz constants yield and the regime avoids lazy training, admitting representation learning. This work provides a rigorous mean-field bridge between finite- MoEs and a continuum limit, suggesting directions for time-uniform bounds and joint depth-width limits.

Abstract

In this work, we study the asymptotic behavior of Mixture of Experts (MoE) trained via gradient flow on supervised learning problems. Our main result establishes the propagation of chaos for a MoE as the number of experts diverges. We demonstrate that the corresponding empirical measure of their parameters is close to a probability measure that solves a nonlinear continuity equation, and we provide an explicit convergence rate that depends solely on the number of experts. We apply our results to a MoE generated by a quantum neural network.
Paper Structure (9 sections, 10 theorems, 76 equations)

This paper contains 9 sections, 10 theorems, 76 equations.

Table of Contents

  1. Introduction
  2. Our results
  3. Preliminaries and notation
  4. Training data
  5. The Wasserstein distance of order 2
  6. Particle systems and propagation of chaos
  7. Mean field limit for general objective functions
  8. Experts given by quantum circuits
  9. Conclusions

Key Result

Theorem 1.1

Consider the MoE $F(\Theta,x)$ induced by the set of $N$ identical experts $\{f(\theta^{i},x):i=1,\ldots,N\}$ where $x$ represents a generic input, $\Theta\coloneqq(\theta^{1},\ldots \theta^{N})$ is the vector of parameters supported on the Torus $\mathbb{T}^{d}$ of dimension $d\in{\mathbb N}$ with Then, there exists a positive constant $C>0$ independent of $N$, and depending on $t$, such that w

Theorems & Definitions (20)

  • Theorem 1.1
  • Theorem 2.1
  • proof
  • Corollary 2.1
  • proof
  • Definition 2.1
  • Theorem 2.2
  • proof
  • Lemma 2.1
  • proof
  • ...and 10 more