Improving Minimax Estimation Rates for Contaminated Mixture of Multinomial Logistic Experts via Expert Heterogeneity

TL;DR

This work develops the first minimax-convergence analysis for a contaminated mixture of multinomial logistic experts in a classification setting, where a frozen pretrained expert is combined with a trainable adapter via softmax gating. By distinguishing homogeneous and heterogeneous expert regimes, it proves identifiability and derives uniform density-estimation rates, alongside matching minimax lower bounds, demonstrating that expert heterogeneity yields faster, near-parametric estimation rates than homogeneity. The theoretical results are complemented by numerical experiments that corroborate the slower rates under homogeneous coupling and the accelerated convergence under heterogeneity, offering guidance for practical fine-tuning of contaminated MoEs. Overall, the findings provide a principled design rule: encourage structural heterogeneity between experts to achieve improved sample efficiency and estimation accuracy in contaminated MoE models.

Abstract

Contaminated mixture of experts (MoE) is motivated by transfer learning methods where a pre-trained model, acting as a frozen expert, is integrated with an adapter model, functioning as a trainable expert, in order to learn a new task. Despite recent efforts to analyze the convergence behavior of parameter estimation in this model, there are still two unresolved problems in the literature. First, the contaminated MoE model has been studied solely in regression settings, while its theoretical foundation in classification settings remains absent. Second, previous works on MoE models for classification capture pointwise convergence rates for parameter estimation without any guaranty of minimax optimality. In this work, we close these gaps by performing, for the first time, the convergence analysis of a contaminated mixture of multinomial logistic experts with homogeneous and heterogeneous structures, respectively. In each regime, we characterize uniform convergence rates for estimating parameters under challenging settings where ground-truth parameters vary with the sample size. Furthermore, we also establish corresponding minimax lower bounds to ensure that these rates are minimax optimal. Notably, our theories offer an important insight into the design of contaminated MoE, that is, expert heterogeneity yields faster parameter estimation rates and, therefore, is more sample-efficient than expert homogeneity.

Figures (3)

• Figure 1: Homogeneous-expert regime. Log–log plots of parameter estimation errors versus the sample size $n$ in the homogeneous setting. Figures \ref{['fig:non_dist_tanh']} and \ref{['fig:non_dist_gelu']} illustrate two homogeneous-expert configurations with shared activations: $\tanh$ in Figure \ref{['fig:non_dist_tanh']} and $\mathop{\mathrm{GELU}}$ in Figure \ref{['fig:non_dist_gelu']}. In each figure, blue dots denote the mean estimation error across different runs at each sample size with vertical error bars indicating one standard deviation, while the orange dashed line shows the fitted power-law trend.
• Figure 2: Heterogeneous-expert regime. Log–log plots of parameter estimation errors versus the sample size $n$ in the heterogeneous setting. Figures \ref{['fig:dist_case_linear_tanh']} and \ref{['fig:dist_case_tanh_linear']} compare two configurations with swapped linear and $\tanh$ activations between the pretrained and adapter experts. The remaining visualization conventions are the same as in Figure \ref{['fig:non_distinguishable_all']}.
• Figure 3: Heterogeneous setting with fixed ground truth parameters. Log--log plots of parameter estimation errors as functions of the sample size $n$. Blue dots denote mean estimation errors across independent runs with error bars indicating one standard deviation, while orange dashed lines represent fitted power-law trends.

Theorems & Definitions (26)

• Proposition 1: Identifiability
• Proposition 2: Density Estimation Rate
• Definition 1: Strong Identifiability
• Theorem 1
• Theorem 2
• Theorem 3
• Theorem 4
• proof
• Lemma 1
• proof