SpecMD: A Comprehensive Study On Speculative Expert Prefetching

TL;DR

This work addresses memory bottlenecks in Mixture-of-Experts inference by introducing SpecMD, a standardized framework to benchmark ad-hoc caching policies across hardware configurations and across policy interactions. It reveals that traditional temporal eviction assumptions fail for MoE access patterns and proposes Least-Stale, a hybrid eviction strategy that balances temporal and spatial locality to dramatically reduce collision misses and improve TTFT. The study also shows that dynamic, score-based prefetching and route-aware policies generally outperform fixed or purely recency-based methods, with policy interactions and hardware constraints shaping optimal configurations. Overall, SpecMD enables reproducible, cross-model, cross-hardware evaluation of MoE caching strategies and demonstrates substantial practical gains through coordinated policy design.

Abstract

Mixture-of-Experts (MoE) models enable sparse expert activation, meaning that only a subset of the model's parameters is used during each inference. However, to translate this sparsity into practical performance, an expert caching mechanism is required. Previous works have proposed hardware-centric caching policies, but how these various caching policies interact with each other and different hardware specification remains poorly understood. To address this gap, we develop \textbf{SpecMD}, a standardized framework for benchmarking ad-hoc cache policies on various hardware configurations. Using SpecMD, we perform an exhaustive benchmarking of several MoE caching strategies, reproducing and extending prior approaches in controlled settings with realistic constraints. Our experiments reveal that MoE expert access is not consistent with temporal locality assumptions (e.g LRU, LFU). Motivated by this observation, we propose \textbf{Least-Stale}, a novel eviction policy that exploits MoE's predictable expert access patterns to reduce collision misses by up to $85\times$ over LRU. With such gains, we achieve over $88\%$ hit rates with up to $34.7\%$ Time-to-first-token (TTFT) reduction on OLMoE at only $5\%$ or $0.6GB$ of VRAM cache capacity.

Figures (9)

• Figure 1: Least-Stale eviction as drop-in improvement across baseline approaches. Our novel eviction policy achieves 10.7--34.7% TTFT reduction on OLMoE-1B-7B at 5% cache capacity (0.6 GB).
• Figure 2: Main(red) and Prefetch (yellow) loops operate synchronously and are directly invoked by the model's after gate. Watchdog (blue) operates asynchronously, constatnly monitoring the prefetch queue.
• Figure 3: Top: LRU eviction as a purely temporal policy. Blocks with higher intensity indicate higher eviction priority; consequently, experts from the immediately following layer are most likely to be evicted. Bottom: Illustration of the Least-Stale (LS) policy, which combines temporal and positional signals. Cached experts are marked as current (green, accessed in the current forward pass) or stale (red, accessed in a previous pass), while layer index encodes position. Higher color intensity corresponds to higher eviction probability. Stale experts have a higher eviction priority than current ones.
• Figure 4: Collision miss across four eviction policies (FLD, SB, LRU, LS) at 5% capacity threshold showing total misses (gray) and collision misses (red). Refer to Section \ref{['sec:eviction']} for abbreviations.
• Figure 5: Per-layer collision miss rates for OLMoE across 16 layers. Least-Stale (red) maintains near-zero collisions across layers.