Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mean-Field Langevin Dynamics for Signed Measures via a Bilevel Approach

Guillaume Wang, Alireza Mousavi-Hosseini, Lénaïc Chizat

TL;DR

This paper extends Mean-Field Langevin Dynamics (MFLD) to convex optimization over signed measures by comparing two reductions to probability measures: lifting and bilevel. It shows that the lifting approach generally fails to satisfy joint displacement smoothness and uniform LSI, while the bilevel reduction preserves these properties under mild assumptions, enabling global convergence through annealing. The authors derive faster, bilevel-specific annealing schedules that achieve a fixed multiplicative accuracy with time complexity that scales more favorably in parameters like the regularization $\lambda$ and dimension $d$, compared to classical annealing. They also establish that for learning a single neuron, the local LSI constant along MFLD-Bilevel can be independent of $\beta$, $\lambda$, and $d$, implying strong local convergence and shedding light on the practical efficiency of the bilevel approach in high-dimensional settings.

Abstract

Mean-field Langevin dynamics (MLFD) is a class of interacting particle methods that tackle convex optimization over probability measures on a manifold, which are scalable, versatile, and enjoy computational guarantees. However, some important problems -- such as risk minimization for infinite width two-layer neural networks, or sparse deconvolution -- are originally defined over the set of signed, rather than probability, measures. In this paper, we investigate how to extend the MFLD framework to convex optimization problems over signed measures. Among two known reductions from signed to probability measures -- the lifting and the bilevel approaches -- we show that the bilevel reduction leads to stronger guarantees and faster rates (at the price of a higher per-iteration complexity). In particular, we investigate the convergence rate of MFLD applied to the bilevel reduction in the low-noise regime and obtain two results. First, this dynamics is amenable to an annealing schedule, adapted from Suzuki et al. (2023), that results in improved convergence rates to a fixed multiplicative accuracy. Second, we investigate the problem of learning a single neuron with the bilevel approach and obtain local exponential convergence rates that depend polynomially on the dimension and noise level (to compare with the exponential dependence that would result from prior analyses).

Mean-Field Langevin Dynamics for Signed Measures via a Bilevel Approach

TL;DR

This paper extends Mean-Field Langevin Dynamics (MFLD) to convex optimization over signed measures by comparing two reductions to probability measures: lifting and bilevel. It shows that the lifting approach generally fails to satisfy joint displacement smoothness and uniform LSI, while the bilevel reduction preserves these properties under mild assumptions, enabling global convergence through annealing. The authors derive faster, bilevel-specific annealing schedules that achieve a fixed multiplicative accuracy with time complexity that scales more favorably in parameters like the regularization and dimension , compared to classical annealing. They also establish that for learning a single neuron, the local LSI constant along MFLD-Bilevel can be independent of , , and , implying strong local convergence and shedding light on the practical efficiency of the bilevel approach in high-dimensional settings.

Abstract

Mean-field Langevin dynamics (MLFD) is a class of interacting particle methods that tackle convex optimization over probability measures on a manifold, which are scalable, versatile, and enjoy computational guarantees. However, some important problems -- such as risk minimization for infinite width two-layer neural networks, or sparse deconvolution -- are originally defined over the set of signed, rather than probability, measures. In this paper, we investigate how to extend the MFLD framework to convex optimization problems over signed measures. Among two known reductions from signed to probability measures -- the lifting and the bilevel approaches -- we show that the bilevel reduction leads to stronger guarantees and faster rates (at the price of a higher per-iteration complexity). In particular, we investigate the convergence rate of MFLD applied to the bilevel reduction in the low-noise regime and obtain two results. First, this dynamics is amenable to an annealing schedule, adapted from Suzuki et al. (2023), that results in improved convergence rates to a fixed multiplicative accuracy. Second, we investigate the problem of learning a single neuron with the bilevel approach and obtain local exponential convergence rates that depend polynomially on the dimension and noise level (to compare with the exponential dependence that would result from prior analyses).

Paper Structure

This paper contains 52 sections, 46 theorems, 248 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Particle methods and mean-field limits.
  4. Multiple kernel learning and bilevel training of NNs.
  5. Background on guarantees for mean-field Langevin dynamics
  6. Reductions from signed measures to probability measures
  7. Reduction by lifting
  8. Issues caused by the disconnectedness of $\Omega^*$.
  9. Incompatibility with MFLD.
  10. Reduction by bilevel optimization
  11. Link between the lifted and bilevel reductions.
  12. Compability with MFLD.
  13. Global convergence and annealing for MFLD-Bilevel
  14. Faster convergence to a fixed multiplicative accuracy
  15. Improved annealing schedule.
  16. ...and 37 more sections

Key Result

Theorem 2.1

Consider $F: \mathcal{P}(\Omega) \to \mathbb{R}$ and $(\mu_t)$ as in eq:bg_MFLD:MFLD_PDE. If P0, P1 and P2 are satisfied then for $t\geq 0$ it holds

Figures (1)

  • Figure 1: The regularized training loss $G_\lambda(\nu)$\ref{['eq:pb-signed-measures']} of a 2NN with the ReLU activation, learning a teacher 2NN with the 4th degree Hermite polynomial as its activation. In both plots, $d=10$ and $\lambda = \beta^{-1} = 10^{-3}$. The implementation details are provided in \ref{['app:experiment']}. Plots are averaged over 5 experiments. $G_\lambda^*$ is the best value achieved at each experiment. In Fig. (\ref{['fig:bileveL_vs_lifting']}), "Conic" refers to using the metric \ref{['eq:reduc:lifting:metric_Omega*']} with $q_r=1,q_w=1$, while "Canonical" refers to the choice of $q_r=2,q_w=0$.

Theorems & Definitions (91)

  • Definition 2.1: Local LSI
  • Theorem 2.1: chizat2022meannitanda2022convex
  • Proposition 3.1
  • Proposition 3.2
  • Remark 3.1
  • Proposition 3.3
  • Proposition 3.4
  • Definition 4.1
  • Proposition 4.1: Baseline "annealing" schedule: constant $\beta_t$
  • Theorem 4.1
  • ...and 81 more