Online Mixture of Experts: No-Regret Learning for Optimal Collective Decision-Making

TL;DR

This work tackles online aggregation of multiple experts to maximize collective accuracy rather than single-expert best-in-hindsight. It introduces two algorithms: Successive Expert Elimination (SEE) for egalitarian voting and a $\theta$-Weighted Majority Voting (WMV) framework for heterogeneous expert competencies, both with no-regret guarantees under full-bandit feedback. The authors formalize the problem with a standardizer, a parametric aggregation $\mathfrak{A}_\theta$, and a scoring function $\mathfrak{G}$, and provide a Mixed-Integer Program (MIP) formulation for optimal WMV weights, including a tight equality constraint $\sum_i \theta_i = 2Q$. Theoretical results include a PAC-style regret bound for SEE $R_T=\mathcal{O}\left( \frac{N}{\tilde{\epsilon}^2}\log T \right)$ and a $\mathcal{O}\left( \sqrt{N T \log T} \right)$ bound for WMV, complemented by tangible LLM ensemble experiments (GSM8K, CommonsenseQA, BoolQ) showing superior performance and online adaptability. The work thus bridges online learning and social choice to enable robust, scalable expert ensembles in modern AI systems, while noting scalability and contextualization limitations as directions for future work.

Abstract

We explore the use of expert-guided bandit learning, which we refer to as online mixture-of-experts (OMoE). In this setting, given a context, a candidate committee of experts must determine how to aggregate their outputs to achieve optimal results in terms of aggregate accuracy. We propose two algorithms to address this problem. The first algorithm combines aggregate voting with UCB-driven successive elimination, efficiently pruning suboptimal exploration actions. The second algorithm employs an online weighted-majority-voting mechanism, leveraging the respective voting power of each expert proportional to their predictive power. We derive theoretical guarantees for the regret properties in the bandit setting under ideal circumstances, and empirical results are provided accordingly. As a modern study on applications, these methods are applied to the online fine-tuning of a set of expert large language models (LLMs), where after each response, the generative LLM dynamically reweighs its set of experts and/or selects the optimal committee of experts to generate the most accurate response. Our results introduce new methodologies and no-regret guarantees for combining multiple experts to improve on the performance of the an aggregate model overall.

Figures (10)

• Figure 1: Online Mixture of Experts via Majority Voting. At each timestep, a context $\texttt{x}$ is sampled from a distribution $P(\texttt{x})$. A prompt derived from $\texttt{x}$ is used to query a set of expert candidates $\mathcal{E}_{\texttt{cand}}$. The responses are aggregated via a majority voting mechanism parameterized by $\theta$, yielding a single output. All expert responses and the aggregated output are evaluated by a scoring function (e.g., an oracle or human). The voting parameters $\theta$ are updated online to maximize expected score, using full-bandit feedback over a finite time horizon.
• Figure 2: Illustration of a breakage event $\texttt{B}_{ij}$.
• Figure 3: Comparison of Cumulative Regret for MoE Voting Environments. Fig. \ref{['fig:see_bern']}: SEE on simulated Bernoulli experts; Fig. \ref{['fig:theta_wmv_bern']}: $\theta$-WMV on simulated Bernoulli experts; Fig. \ref{['fig:llm_experiments']}: LLM experts across domains (e.g., GSM8k, BoolQ etc.), with tasks sampled from each domain. Shaded regions show $\pm \mathsf{UCB}$ uncertainty. Evidently, SEE and $\theta$-WMV consistently outperform baselines in regret minimization.
• Figure 4: Visualization of the performance comparison of language models across three benchmarks (GSM8K, CQA, BoolQ) as a function of parameter count (7B, 8B, 14B).
• Figure : Config. SE1 & SC1 - $p_i$: $[0.1, 0.65, 0.77, 0.79, 0.8]$.

Theorems & Definitions (37)

• Remark 1
• Remark 2
• Lemma 2.1
• Lemma 2.2
• Proposition 2.1
• Lemma 2.3
• Definition 3.1
• Definition 3.2
• Lemma 3.1
• Theorem 1