LaMoSys3.5D: Enabling 3.5D-IC-Based Large Language Model Inference Serving Systems via Hardware/Software Co-Design

TL;DR

LaMoSys3.5D tackles end-to-end LLM inference serving by proposing a heterogeneous 3.5D-IC architecture that disaggregates prefill and decode workloads across compute- and memory-rich chiplets on a 2.5D interposer. It introduces a 3D-native $D^3$ dataflow, mesh-aware parallel mapping, and dynamic scheduling within a temperature-aware end-to-end simulation and hierarchical design-space exploration framework. Across diverse LLMs and workloads, it achieves up to 62% throughput-per-watt improvements over DGX-A100 and up to 4.87× better end-to-end latency versus prior 3D designs, with strong decode acceleration on long outputs. The work also provides concrete design guidelines for 3.5D-IC architectures and end-to-end inference serving, highlighting the importance of thermal considerations in such stacks.

Abstract

The success of large language models LLMs amplifies the need for highthroughput energyefficient inference at scale. 3DDRAMbased accelerators provide high memory bandwidth and therefore an opportunity to accelerate the bandwidthbound decode phase. However, how to adequately balance compute density for prefill with bandwidthcapacity for decode remains open. Moreover, most prior designs do not target endtoend serving, leaving the codesign of dataflow, parallel mapping, and scheduling underexplored. To bridge the gap, we present LaMoSys3.5D, to our knowledge the first scalable 3.5DIC architecture for LLM serving. LaMoSys3.5D composes heterogeneous 3DDRAM chiplets on a 2.5D interposer: computerich chiplets for prefill and bandwidthcapacityrich chiplets for decode. To realize efficient serving, we adopt a hardwaresoftware codesign spanning dataflow, parallel mapping, and introduce a thermalaware modeling and hierarchical designspace exploration framework. Across diverse LLMs and workloads, LaMoSys3.5D improves throughputperwatt over DGXA100 systems by 62 and achieves a 4.87 better endtoend latency geomean versus prior 3D designs. We further distill intriguing design guidelines for 3.5DIC architectures and endtoend inference serving.

Figures (15)

• Figure 1: LLM inference flow, including (a) the prefill and decode phases of the input sequence "Large Language Model", (b) architecture blocks of each transformer layer, (c) computation diagram of the self-attention block, (d) variants of attention operators, (e) variants of dense FFN operations.
• Figure 2: AI and memory requirements for the prefill and decode phases of different LLMs. The batch size of decode ranges from 1 to 64, indicated by the size of the triangles.
• Figure 3: (a) Layer-wise 3D temperature map for our 3.5D-IC running certain workload. (b) Total power (7 nm logic and 22 nm DRAM) and chiplet bandwidth versus temperature. (c) Temperature-aware roofline model.
• Figure 4: Hardware template of the proposed LaMoSys3.5D.
• Figure 5: (a) Intra-PE data path and executions; (b) representative PP/TP mapping scheme with (TP, PP) = (8, 1) and (4, 2) for prefill and decode, respectively; (c) Timing of operations of one stage executed on PC.