Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Geometric Regularization in Mixture-of-Experts: The Disconnect Between Weights and Activations

Hyunjun Kim

TL;DR

This work tests whether enforcing weight-space orthogonality via a geometric regularizer improves diversity and perplexity in Mixture-of-Experts (MoE) models. By defining an explicit orthogonality loss and two Mean Squared Overlap (MSO) measures for weights and activations, the authors evaluate seven regularization strengths on NanoGPT-MoE across TinyStories, WikiText-103, and PTB. They find a fundamental weight-activation disconnect: weight MSO is not reduced (and can increase by up to 114%), while activation MSO remains ≈0.57 regardless of regularization, and perplexity changes are dataset-dependent with high variance. The non-linearities (e.g., SiLU) and input distribution dynamics appear to compress activation differences, undermining the geometric regularization. Consequently, weight-space regularization is unreliable for promoting MoE diversity, suggesting a shift toward activation-space constraints or alternative diversity objectives for robust MoE design.

Abstract

Mixture-of-Experts (MoE) models achieve efficiency through sparse activation, but the role of geometric regularization in expert specialization remains unclear. We apply orthogonality loss to enforce expert diversity and find it fails on multiple fronts: it does not reduce weight-space overlap (MSO actually increases by up to 114%), activation-space overlap remains high (~0.6) regardless of regularization, and effects on performance are inconsistent -- marginal improvement on WikiText-103 (-0.9%), slight degradation on TinyStories (+0.9%), and highly variable results on PTB (std > 1.0). Our analysis across 7 regularization strengths reveals no significant correlation (r = -0.293, p = 0.523) between weight and activation orthogonality. These findings demonstrate that weight-space regularization neither achieves its geometric goal nor reliably improves performance, making it unsuitable for MoE diversity.

Geometric Regularization in Mixture-of-Experts: The Disconnect Between Weights and Activations

TL;DR

This work tests whether enforcing weight-space orthogonality via a geometric regularizer improves diversity and perplexity in Mixture-of-Experts (MoE) models. By defining an explicit orthogonality loss and two Mean Squared Overlap (MSO) measures for weights and activations, the authors evaluate seven regularization strengths on NanoGPT-MoE across TinyStories, WikiText-103, and PTB. They find a fundamental weight-activation disconnect: weight MSO is not reduced (and can increase by up to 114%), while activation MSO remains ≈0.57 regardless of regularization, and perplexity changes are dataset-dependent with high variance. The non-linearities (e.g., SiLU) and input distribution dynamics appear to compress activation differences, undermining the geometric regularization. Consequently, weight-space regularization is unreliable for promoting MoE diversity, suggesting a shift toward activation-space constraints or alternative diversity objectives for robust MoE design.

Abstract

Mixture-of-Experts (MoE) models achieve efficiency through sparse activation, but the role of geometric regularization in expert specialization remains unclear. We apply orthogonality loss to enforce expert diversity and find it fails on multiple fronts: it does not reduce weight-space overlap (MSO actually increases by up to 114%), activation-space overlap remains high (~0.6) regardless of regularization, and effects on performance are inconsistent -- marginal improvement on WikiText-103 (-0.9%), slight degradation on TinyStories (+0.9%), and highly variable results on PTB (std > 1.0). Our analysis across 7 regularization strengths reveals no significant correlation (r = -0.293, p = 0.523) between weight and activation orthogonality. These findings demonstrate that weight-space regularization neither achieves its geometric goal nor reliably improves performance, making it unsuitable for MoE diversity.
Paper Structure (38 sections, 4 equations, 1 figure, 4 tables)

This paper contains 38 sections, 4 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Hypothesis.
  3. Finding.
  4. Why?
  5. Contributions.
  6. Method
  7. Orthogonality Loss.
  8. Mean Squared Overlap (MSO).
  9. Activation MSO.
  10. Experiments
  11. Setup.
  12. Implementation Details.
  13. Weight MSO Under Regularization
  14. Perplexity Does Not Improve
  15. Why Are p-values High?
  16. ...and 23 more sections

Figures (1)

  • Figure 1: Weight-Activation Gap. Weight MSO responds to regularization; activation MSO does not. No significant correlation ($r = -0.293$, $p = 0.523$).