Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Taming the Tail: NoI Topology Synthesis for Mixed DL Workloads on Chiplet-Based Accelerators

Arnav Shukla, Harsh Sharma, Srikant Bharadwaj, Vinayak Abrol, Sujay Deb

TL;DR

The paper tackles tail-latency challenges in NoI topologies for chiplet-based accelerators under memory-driven MoE workloads. It introduces an Interference Score to quantify worst-case slowdown under contention and formulates NoI synthesis as a multi-objective optimization, solved by PARL, a partition-aware reinforcement learner. PARL-generated topologies achieve improved tail robustness by creating memory-cut islands and partitioned pathways, reducing worst-case slowdown to roughly 1.2x while maintaining competitive mean throughput compared to dense meshes. The results underscore the need for workload-aware, non-uniform NoI designs to meet SLAs in heterogeneous chiplet systems, informing future interconnect architectures for AI workloads.

Abstract

Heterogeneous chiplet-based systems improve scaling by disag-gregating CPUs/GPUs and emerging technologies (HBM/DRAM).However this on-package disaggregation introduces a latency inNetwork-on-Interposer(NoI). We observe that in modern large-modelinference, parameters and activations routinely move backand forth from HBM/DRAM, injecting large, bursty flows into theinterposer. These memory-driven transfers inflate tail latency andviolate Service Level Agreements (SLAs) across k-ary n-cube base-line NoI topologies. To address this gap we introduce an InterferenceScore (IS) that quantifies worst-case slowdown under contention.We then formulate NoI synthesis as a multi-objective optimization(MOO) problem. We develop PARL (Partition-Aware ReinforcementLearner), a topology generator that balances throughput, latency,and power. PARL-generated topologies reduce contention at the memory cut, meet SLAs, and cut worst-case slowdown to 1.2 times while maintaining competitive mean throughput relative to link-rich meshes. Overall, this reframes NoI design for heterogeneouschiplet accelerators with workload-aware objectives.

Taming the Tail: NoI Topology Synthesis for Mixed DL Workloads on Chiplet-Based Accelerators

TL;DR

The paper tackles tail-latency challenges in NoI topologies for chiplet-based accelerators under memory-driven MoE workloads. It introduces an Interference Score to quantify worst-case slowdown under contention and formulates NoI synthesis as a multi-objective optimization, solved by PARL, a partition-aware reinforcement learner. PARL-generated topologies achieve improved tail robustness by creating memory-cut islands and partitioned pathways, reducing worst-case slowdown to roughly 1.2x while maintaining competitive mean throughput compared to dense meshes. The results underscore the need for workload-aware, non-uniform NoI designs to meet SLAs in heterogeneous chiplet systems, informing future interconnect architectures for AI workloads.

Abstract

Heterogeneous chiplet-based systems improve scaling by disag-gregating CPUs/GPUs and emerging technologies (HBM/DRAM).However this on-package disaggregation introduces a latency inNetwork-on-Interposer(NoI). We observe that in modern large-modelinference, parameters and activations routinely move backand forth from HBM/DRAM, injecting large, bursty flows into theinterposer. These memory-driven transfers inflate tail latency andviolate Service Level Agreements (SLAs) across k-ary n-cube base-line NoI topologies. To address this gap we introduce an InterferenceScore (IS) that quantifies worst-case slowdown under contention.We then formulate NoI synthesis as a multi-objective optimization(MOO) problem. We develop PARL (Partition-Aware ReinforcementLearner), a topology generator that balances throughput, latency,and power. PARL-generated topologies reduce contention at the memory cut, meet SLAs, and cut worst-case slowdown to 1.2 times while maintaining competitive mean throughput relative to link-rich meshes. Overall, this reframes NoI design for heterogeneouschiplet accelerators with workload-aware objectives.

Paper Structure

This paper contains 13 sections, 5 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Inevitability of Irregular & Bursty Traffic
  4. Irregular HBM Traffic
  5. Catastrophic Interference at the Memory Cut
  6. The Case for Targeted Link Augmentation
  7. Proposed Partition-Aware NoI Design
  8. PARL Topology Generator
  9. Evaluation
  10. Empirical Validation of Theoretical Claims
  11. Baseline Topology Performance Under Mixed Workloads
  12. PARL-Generated Topologies for Robust Isolation
  13. Conclusion

Figures (5)

  • Figure 1: Performance analysis of baseline and augmented topologies under mixed workloads. (a) All original baselines suffer $4\times$–$5\times$ throughput degradation, violating SLA thresholds. (b) Interference shows no correlation with raw link count ($R^2 \approx 0.15$), indicating that bandwidth alone is insufficient. (c) Per-expert slowdown heatmap reveals severe, unbalanced SLA violations.
  • Figure 2: Topology Comparison: Blue=Common, Green=RL-only, Red=Mesh-only
  • Figure 3: Latency and Throughput of baseline and PARL topology
  • Figure 4: Ingress traffic at HBM/IOD nodes showing bursty peaks during MoE gating events, validating Claim A.
  • Figure 5: Memory-adjacent cut ($C_M$) as system bottleneck with tail latency inflation under load, validating Claim B.

Theorems & Definitions (4)

  • Definition 1: Expert Structure and MoE Routing
  • Definition 2: Regular NoI Topology
  • Definition 3: Memory Cut
  • Definition 4: Arrival and Service Variability