Convergence Dynamics of Over-Parameterized Score Matching for a Single Gaussian

TL;DR

This work analyzes gradient descent for an over-parameterized score-matching model learning a single Gaussian in diffusion-model settings. It proves global convergence in the high-noise regime and reveals rich, initialization-dependent dynamics in the low-noise regime, including a scenario where all parameters converge under exponentially small initialization and a contrasting regime where only one parameter converges under random initialization with vanishing loss. The results connect score matching with gradient EM in the over-parameterized context and establish rate bounds, including an O(1/τ) decay and near-matching lower bounds, highlighting the critical roles of noise level and initialization in training diffusion-based models. The findings offer guidance for training strategies and point to future work on time-averaged losses and stochastic optimization in over-parameterized score-based models.

Abstract

Score matching has become a central training objective in modern generative modeling, particularly in diffusion models, where it is used to learn high-dimensional data distributions through the estimation of score functions. Despite its empirical success, the theoretical understanding of the optimization behavior of score matching, particularly in over-parameterized regimes, remains limited. In this work, we study gradient descent for training over-parameterized models to learn a single Gaussian distribution. Specifically, we use a student model with $n$ learnable parameters and train it on data generated from a single ground-truth Gaussian using the population score matching objective. We analyze the optimization dynamics under multiple regimes. When the noise scale is sufficiently large, we prove a global convergence result for gradient descent. In the low-noise regime, we identify the existence of a stationary point, highlighting the difficulty of proving global convergence in this case. Nevertheless, we show convergence under certain initialization conditions: when the parameters are initialized to be exponentially small, gradient descent ensures convergence of all parameters to the ground truth. We further prove that without the exponentially small initialization, the parameters may not converge to the ground truth. Finally, we consider the case where parameters are randomly initialized from a Gaussian distribution far from the ground truth. We prove that, with high probability, only one parameter converges while the others diverge, yet the loss still converges to zero with a $1/τ$ rate, where $τ$ is the number of iterations. We also establish a nearly matching lower bound on the convergence rate in this regime. This is the first work to establish global convergence guarantees for Gaussian mixtures with at least three components under the score matching framework.

Figures (1)

• Figure 1: Experimental results illustrating two convergence behaviors. (a, c) For $M=4$, all parameters converge to the ground truth, consistent with \ref{['sec:all_converge']}. (b, d) For $M=6$, only one parameter converges while the others diverge, consistent with \ref{['sec:random_init']}. The loss curve in (d) exhibits a convergence rate close to $O(1/\tau)$ in the final phase.

Theorems & Definitions (61)

• Theorem 2.1
• Lemma 2.2: Part of \ref{['lm:gradient']}
• Theorem 3.1
• Theorem 3.2
• Lemma 3.3: See also \ref{['lm:gradient_near']}
• Theorem 3.4
• Theorem 3.5
• Corollary 3.6
• Theorem 3.7
• Lemma A.1: Gaussian Tail Bound