Stratum: System-Hardware Co-Design with Tiered Monolithic 3D-Stackable DRAM for Efficient MoE Serving

TL;DR

Stratum tackles the memory bandwidth and data-movement bottlenecks in MoE serving for large language models by proposing a system–hardware co-design that integrates Monolithic 3D-Stackable DRAM with a dedicated near-memory processor and GPU on a 2.5D interposer. It introduces in-memory tiering to exploit latency variation across thousands of Mono3D DRAM layers and a topic-aware data-placement strategy guided by an offline-profiling-based expert hit rate, using a lightweight topic classifier to drive scheduling. The approach demonstrates up to 8.29x decoding throughput and 7.66x energy efficiency improvements over GPU baselines across representative MoE models, validating a scalable path for efficient MoE serving. These results highlight the practical potential of Mono3D DRAM-based near-memory computing for cost-effective, high-throughput LLM inference at scale.

Abstract

As Large Language Models (LLMs) continue to evolve, Mixture of Experts (MoE) architecture has emerged as a prevailing design for achieving state-of-the-art performance across a wide range of tasks. MoE models use sparse gating to activate only a handful of expert sub-networks per input, achieving billion-parameter capacity with inference costs akin to much smaller models. However, such models often pose challenges for hardware deployment due to the massive data volume introduced by the MoE layers. To address the challenges of serving MoE models, we propose Stratum, a system-hardware co-design approach that combines the novel memory technology Monolithic 3D-Stackable DRAM (Mono3D DRAM), near-memory processing (NMP), and GPU acceleration. The logic and Mono3D DRAM dies are connected through hybrid bonding, whereas the Mono3D DRAM stack and GPU are interconnected via silicon interposer. Mono3D DRAM offers higher internal bandwidth than HBM thanks to the dense vertical interconnect pitch enabled by its monolithic structure, which supports implementations of higher-performance near-memory processing. Furthermore, we tackle the latency differences introduced by aggressive vertical scaling of Mono3D DRAM along the z-dimension by constructing internal memory tiers and assigning data across layers based on access likelihood, guided by topic-based expert usage prediction to boost NMP throughput. The Stratum system achieves up to 8.29x improvement in decoding throughput and 7.66x better energy efficiency across various benchmarks compared to GPU baselines.

Figures (18)

• Figure 1: Architectures of dense transformer-based LLM (left) and Mixture of Experts (MoE) LLM (right).
• Figure 2: Monolithic 3D-Stackable DRAM with vertically stacked horizontal 1T1C DRAM cells. Bitlines are vertically routed to avoid sense margin variations, and wordlines are routed through staircases. The activation latency varies by layers due to wordline staircases.
• Figure 3: HBM versus Mono3D DRAM on 2.5D integration platform with a xPU die. The HBM and Mono3D DRAM are attached to the logic base die through TSVs and Cu-Cu hybrid bonding, respectively.
• Figure 4: Expert hit profiling from LLaMA-4 Scout (16 Experts).
• Figure 5: Example Stratum configurations.