Understanding Expert Structures on Minimax Parameter Estimation in Contaminated Mixture of Experts

TL;DR

The paper tackles minimax-rate estimation for contaminated MoE models where a fixed pre-trained component is augmented by a trainable prompt. By introducing a distinguishability condition and exhaustively analyzing linear and nonlinear expert structures under Gaussian and non-Gaussian pre-trained densities, it derives uniform convergence rates and matching minimax lower bounds for the mixing proportion and prompt parameters. It reveals how prompt vanishing and prompt merging shape estimation difficulties, and provides practical guidelines for prompt design to optimize fine-tuning efficiency. Empirical results corroborate the theory and demonstrate how structural differences between experts can improve identifiability and convergence in large-scale model adaptation tasks.

Abstract

We conduct the convergence analysis of parameter estimation in the contaminated mixture of experts. This model is motivated from the prompt learning problem where ones utilize prompts, which can be formulated as experts, to fine-tune a large-scale pre-trained model for learning downstream tasks. There are two fundamental challenges emerging from the analysis: (i) the proportion in the mixture of the pre-trained model and the prompt may converge to zero during the training, leading to the prompt vanishing issue; (ii) the algebraic interaction among parameters of the pre-trained model and the prompt can occur via some partial differential equations and decelerate the prompt learning. In response, we introduce a distinguishability condition to control the previous parameter interaction. Additionally, we also investigate various types of expert structure to understand their effects on the convergence behavior of parameter estimation. In each scenario, we provide comprehensive convergence rates of parameter estimation along with the corresponding minimax lower bounds. Finally, we run several numerical experiments to empirically justify our theoretical findings.

Figures (6)

• Figure 1: Log-log graphs depicting the empirical convergence rates of the MLE $(\widehat{\lambda}_{n},\widehat{a}_{n},\widehat{b}_{n},\widehat{\nu}_{n})$ to the ground-truth values $(\lambda^*,a^{*},b^{*},\nu^{*})$. The blue lines display the parameter estimation errors, while the red dashed dotted lines are the fitted lines, highlighting the empirical MLE convergence rates. Figure \ref{['fig:thm2-fixed']} and Figure \ref{['fig:thm2-var']} illustrates results for the cases when $\lambda^* = 0.5$ and $\lambda^* = 0.5~n^{-1/4}$, respectively.
• Figure 2: (Theorem \ref{['theorem:sigma-linear-f0-Gaussian-varphi-nonlinear']}:$f_0$ is Gaussian, $\sigma(z)=z$, $\varphi(z)=1/(1+e^{-z})$.) Log-log graphs depicting the empirical convergence rates of the MLE $(\widehat{\lambda}_{n},\widehat{a}_{n},\widehat{b}_{n},\widehat{\nu}_{n})$ to the ground-truth values $(\lambda^*,a^{*},b^{*},\nu^{*})$. The blue lines display the parameter estimation errors, while the red dashed dotted lines are the fitted lines, highlighting the empirical MLE convergence rates. Figure \ref{['fig:thm6-fixed']} and Figure \ref{['fig:thm6-var']} illustrates results for the cases when $\lambda^* = 0.5$ and $\lambda^* = 0.5~n^{-1/4}$, respectively.
• Figure 3: (Theorem \ref{['theorem:sigma-nonlinear-f0-notGaussian']}:$f_0$ is Student's t-distribution, $\sigma(z)=1/(1+e^{-z})$, $\varphi(z)=z$. ) Log-log graphs depicting the empirical convergence rates of the MLE $(\widehat{\lambda}_{n},\widehat{a}_{n},\widehat{b}_{n},\widehat{\nu}_{n})$ to the ground-truth values $(\lambda^*,a^{*},b^{*},\nu^{*})$. The blue lines display the parameter estimation errors, while the red dashed dotted lines are the fitted lines, highlighting the empirical MLE convergence rates. Figure \ref{['fig:thm7-fixed']} and Figure \ref{['fig:thm7-var']} illustrates results for the cases when $\lambda^* = 0.5$ and $\lambda^* = 0.5~n^{-1/4}$, respectively.
• Figure 4: (Theorem \ref{['theorem:sigma-nonlinear-f0-Gaussian-varphi-linear']}:$f_0$ is Gaussian, $\sigma(z)=1/(1+e^{-z})$, $\varphi(z)=z$. ) Log-log graphs depicting the empirical convergence rates of the MLE $(\widehat{\lambda}_{n},\widehat{a}_{n},\widehat{b}_{n},\widehat{\nu}_{n})$ to the ground-truth values $(\lambda^*,a^{*},b^{*},\nu^{*})$. The blue lines display the parameter estimation errors, while the red dashed dotted lines are the fitted lines, highlighting the empirical MLE convergence rates. Figure \ref{['fig:thm8-fixed']} and Figure \ref{['fig:thm8-var']} illustrates results for the cases when $\lambda^* = 0.5$ and $\lambda^* = 0.5~n^{-1/4}$, respectively.
• Figure 5: (Theorem \ref{['theorem:sigma-linear-f0-Gaussian-varphi-linear']}:$f_0$ is Gaussian, $\sigma(z)=\varphi(z)=z$.) Log-log graphs depicting the empirical convergence rates of the MLE $(\widehat{\lambda}_{n},\widehat{a}_{n},\widehat{b}_{n},\widehat{\nu}_{n})$ to the ground-truth values $(\lambda^*,a^{*},b^{*},\nu^{*})$. The blue lines display the parameter estimation errors, while the red dashed dotted lines are the fitted lines, highlighting the empirical MLE convergence rates. Figure \ref{['fig:thm4-case1']} and Figure \ref{['fig:thm4-case2']} illustrates results for Case (i) and Case (ii), respectively.

Theorems & Definitions (45)

• Definition 1: Distinguishability
• Proposition 1: Identifiability
• Theorem 1
• Proposition 2
• Theorem 2: MLE rates
• Theorem 3: Minimax lower bounds
• Theorem 4: MLE rates
• Theorem 5: Minimax lower bounds
• Theorem 6: MLE rates
• Theorem 7: MLE rates