Horseshoe Mixtures-of-Experts (HS-MoE)
Nick Polson, Vadim Sokolov
TL;DR
Horseshoe-MoE (HS-MoE) advances sparse expert routing by placing adaptive global-local shrinkage on gating while representing uncertainty over allocations. It combines Gaussian linear experts with a stick-breaking Pólya–Gamma gated router and a sequential particle-learning algorithm that updates sufficient statistics online, yielding closed-form predictive updates for experts and tractable gating updates. The work provides a principled Bayesian router compatible with Transformer MoE layers, enables marginal-likelihood–based model selection, and connects to universal approximation and sparse generalization bounds, implying improved efficiency and robustness in sparse, streaming settings. Practically, HS-MoE offers uncertainty-aware routing suitable for streaming data and large-scale, sparsely activated expert pools in modern LLMs, with a blueprint for scalable approximations in deployment.
Abstract
Horseshoe mixtures-of-experts (HS-MoE) models provide a Bayesian framework for sparse expert selection in mixture-of-experts architectures. We combine the horseshoe prior's adaptive global-local shrinkage with input-dependent gating, yielding data-adaptive sparsity in expert usage. Our primary methodological contribution is a particle learning algorithm for sequential inference, in which the filter is propagated forward in time while tracking only sufficient statistics. We also discuss how HS-MoE relates to modern mixture-of-experts layers in large language models, which are deployed under extreme sparsity constraints (e.g., activating a small number of experts per token out of a large pool).