Certain Head, Uncertain Tail: Expert-Sample for Test-Time Scaling in Fine-Grained MoE

TL;DR

This work addresses test-time scaling for large language models by exploiting the routing space of fine-grained Mixture-of-Experts (MoE). It identifies a structural pattern in router scores—a certain head of a few high-confidence experts and a long, uniform tail—that can separately support stable, core reasoning and diverse exploration. The authors propose Expert-Sample, a training-free method that deterministically preserves the top-$k_{ ext{keep}}$ experts and probabilistically samples from the tail using temperature-controlled Gumbel-softmax, yielding diverse yet stable outputs without architectural changes. Extensive experiments across multiple fine-grained MoE models and tasks show consistent improvements in pass@N and verification-based accuracy, while incurring negligible overhead. The approach offers a practical, plug-in solution to decouple stability and diversity at the routing level, complementing token-level sampling and verification methods, and enabling more efficient inference-time scaling.

Abstract

Test-time scaling improves LLM performance by generating multiple candidate solutions, yet token-level sampling requires temperature tuning that trades off diversity against stability. Fine-grained MoE, featuring hundreds of well-trained experts per layer and multi-expert activation per token, offers an unexplored alternative through its rich routing space. We empirically characterize fine-grained MoE routing and uncover an informative pattern: router scores exhibit a certain head of high-confidence experts followed by an uncertain tail of low-confidence candidates. While single-run greedy accuracy remains stable when fewer experts are activated, multi-sample pass@n degrades significantly-suggesting that the certain head governs core reasoning capability while the uncertain tail correlates with reasoning diversity. Motivated by these findings, we propose Expert-Sample, a training-free method that preserves high-confidence selections while injecting controlled stochasticity into the uncertain tail, enabling diverse generation without destabilizing outputs. Evaluated on multiple fine-grained MoE models across math, knowledge reasoning, and code tasks, Expert-Sample consistently improves pass@n and verification-based accuracy. On Qwen3-30B-A3B-Instruct evaluated on GPQA-Diamond with 32 parallel samples, pass@32 rises from 85.4% to 91.9%, and accuracy improves from 59.1% to 62.6% with Best-of-N verification.

Figures (13)

• Figure 1: Overview of Expert-Sample. Left: Illustration of the Expert-Sample mechanism with an example from MATH-500, showing how Expert-Sample achieves structural diversity to discover the correct answer, while standard token-sample produces only superficial diversity. Right: Pass@n scaling improvements (upper) and accuracy gains with Best-of-N verification (lower) on Qwen3-30B-A3B-Instruct.
• Figure 2: Empirical findings and motivation for Expert-Sample. (a) Greedy decoding accuracy remains stable when reducing activated experts to half of the default top-k. (b) Pass@n accuracy degrades substantially when expert count is reduced, suggesting the uncertain tail is critical for diverse exploration. (c) Router score distribution for the top 32 ranked positions reveals a certain head with high-confidence experts and an uncertain tail with uniform scores (full distribution is in Appendix \ref{['app:weight_distribution']}). All Qwen3 models shown are Instruct versions.
• Figure 3: Validation results on AIME-120 across three difficulty tiers. For Qwen3-30B-A3B (Instruct), the Correct/Medium/Hard sets contain 79/28/13 problems respectively; for Ling-Lite-1.5, 31/42/47 problems respectively.
• Figure 4: Pass@n accuracy scaling curves on AIME-120 (top), GPQA-Diamond (middle), and LiveCodeBench-V6-Lite (bottom). Shaded regions indicate standard deviation. Expert-Sample consistently outperforms token sampling baselines across all models and tasks.
• Figure 5: Effect of $k_{\mathrm{keep}}$ on Qwen3-30B-A3B (Instruct). Left: Pass@$n$ accuracy on the Uncertain Set across different $k_{\mathrm{keep}}$ values. The shaded region indicates the recommended range ($k/2 \leq k_{\mathrm{keep}} < k$) where Expert-Sample consistently outperforms the baseline. Right: Pass rate on the Correct Set. Stability degrades significantly when $k_{\mathrm{keep}} < k/2$, but remains comparable to baseline once $k_{\mathrm{keep}} \geq k/2$.