ButterflyMoE: Sub-Linear Ternary Experts via Structured Butterfly Orbits

TL;DR

ButterflyMoE tackles the memory bottleneck of Mixture-of-Experts on edge devices by reparameterizing N experts as orbital rotations of a shared ternary substrate, instead of storing independent weight matrices. The method uses Butterfly matrices to construct per-expert rotations with $O(d \log d)$ parameters, yielding an overall memory of $O(d^2 + N \cdot d \log d)$ and enabling sub-linear scaling relative to traditional MoEs. Training jointly optimizes the substrate and the rotations, with per-expert input rotations suppressing quantization outliers and a top-$k$ gating mechanism preserving specialization. Empirically, ButterflyMoE achieves roughly 150× compression at 256 experts and allows 64-expert models to fit on a 4 GB device (e.g., Jetson Nano) with only modest accuracy loss, while maintaining distinct expert behaviors; this paves the way for large-scale, edge-deployed MoEs. Potential limitations include slower inference compared to dense baselines and the need to scale the approach to billion-parameter regimes, but the approach demonstrates a practical path to extreme memory efficiency via geometric parameter-sharing and structured rotations.

Abstract

Linear memory scaling stores $N$ independent expert weight matrices requiring $\mathcal{O}(N \cdot d^2)$ memory, which exceeds edge devices memory budget. Current compression methods like quantization, pruning and low-rank factorization reduce constant factors but leave the scaling bottleneck unresolved. We introduce ButterflyMoE, a method that treats experts not as independent weight matrices but as geometric reorientations of a unified shared quantized substrate. Diversity among experts arises from viewing different angles of shared capacity, not from redundant storage. By applying learned rotations to a shared ternary prototype, each expert yields $\mathcal{O}(d^2 + N \cdot d \log d)$ memory -- sub-linear in the number of experts. The key insight: training these rotations with quantization reduces activation outliers and stabilizes extreme low bit training, where static methods collapse. Across language modeling benchmarks, ButterflyMoE achieves 150 times memory reduction at 256 experts with negligible accuracy loss. This allows 64 experts to fit on 4GB devices compared to standard MoE's 8 experts, showing geometric parametrization breaks linear scaling.

Figures (5)

• Figure 1: The proposed Butterfly MoE layer replaces the standard FFN within a Transformer block
• Figure 2: Top-$k$ gating instantiates experts via lightweight rotations of a shared ternary base matrix $\mathbf{W}_{\text{base}}$ achieving $\mathcal{O}(d \log d)$ parameters per expert and weighted sum aggregation.
• Figure 3: Figure \ref{['fig:memory']}: Memory consumption as a function of the number of experts for standard MoE and ButterflyMoE ($d{=}512$). The y-axis reports total parameter memory in MB; lower is better.
• Figure 4: Quantization stability via learned rotations. (Top left) Untrained substrate weight distribution spreads across $[-4, +4]$. (Top right) After training, weights cluster near ternary grid $\{-1, 0, +1\}$. (Bottom left) Trained weights (green) align with quantization levels (red dashed lines). (Bottom right) Mean squared error: 51.3% (untrained) → 1.43% (trained), a 97.2% reduction.
• Figure 5: Expert output similarity. Pairwise cosine similarity matrix between expert outputs on the WikiText-103 validation set.

Theorems & Definitions (3)

• Proposition 1: Memory Scaling
• Proposition 2: Asymptotic Compression
• Proposition 3: Computational Complexity