Reliability Under Randomness: An Empirical Analysis of Sparse and Dense Language Models Across Decoding Temperatures
Kabir Grover
TL;DR
This study interrogates whether sparse Mixture-of-Experts (MoE) models are less reliable than dense models when decoding is stochastic via temperature scaling. By testing three models (OLMoE-7B, Mixtral-8x7B, Qwen2.5-3B) on deterministic arithmetic tasks across four temperatures, the authors distinguish the effects of sparsity from instruction tuning. The central finding is that instruction tuning, not architectural sparsity, governs robustness to decoding randomness: the sparse instruction-tuned Mixtral matches the dense instruction-tuned Qwen in reliability, while the sparse base OLMoE degrades with higher temperature. These results imply that well-aligned sparse MoE models can be deployed without sacrificing output stability, though base, non-aligned models remain vulnerable to stochastic decoding.
Abstract
The increasing prevalence of sparse Mixture-of-Experts (MoE) architectures in large language models raises important questions regarding their reliability under stochastic decoding. While conditional computation enables substantial gains in computational efficiency, it remains unclear whether the interaction between sparse routing and temperature-based sampling compromises output stability relative to dense architectures. This work investigates whether conditional computation in MoE models amplifies decoding-induced randomness, leading to reduced reliability as temperature increases. We evaluate three representative models: OLMoE-7B (sparse base), Mixtral-8x7B (sparse instruction-tuned), and Qwen2.5-3B (dense instruction-tuned) on deterministic arithmetic reasoning tasks with objectively verifiable answers. Experiments span four decoding configurations, ranging from greedy decoding to T=1.0. Our evaluation encompasses accuracy, format compliance, output consistency across repeated generations, and confidence metrics, totaling 9,360 model generations. Results demonstrate that the sparse instruction-tuned model exhibits stability comparable to the dense instruction-tuned model across all decoding temperatures, while the sparse base model shows systematic degradation as temperature increases. These findings indicate that instruction tuning, rather than architectural sparsity, is the primary determinant of robustness to decoding randomness on deterministic tasks. We discuss the implications of these results for deploying sparse language models in reliability-critical applications, highlighting scenarios in which sparse architectures can be safely adopted without sacrificing output stability.