Heterogeneous Computing: The Key to Powering the Future of AI Agent Inference

TL;DR

The paper tackles memory and bandwidth bottlenecks in AI agent inference by introducing Operational Intensity ($OI$) and Capacity Footprint ($CF$) as memory-centric metrics that extend beyond traditional roofline analyses. It shows how agent workloads, model architectures, and optimizations shift inference regimes and highlight the memory capacity wall, advocating for disaggregated, heterogeneous hardware and optical I/O to sustain scaling. Through analysis of diverse agent workflows and base-model variations, the work argues for per-phase accelerators, hardware-aware co-design, and memory disaggregation as practical strategies. The proposed hypotheses describe a future where system-level heterogeneity, scalable networking, and co-design enable efficient long-context, multi-turn AI agents at scale.

Abstract

AI agent inference is driving an inference heavy datacenter future and exposes bottlenecks beyond compute - especially memory capacity, memory bandwidth and high-speed interconnect. We introduce two metrics - Operational Intensity (OI) and Capacity Footprint (CF) - that jointly explain regimes the classic roofline analysis misses, including the memory capacity wall. Across agentic workflows (chat, coding, web use, computer use) and base model choices (GQA/MLA, MoE, quantization), OI/CF can shift dramatically, with long context KV cache making decode highly memory bound. These observations motivate disaggregated serving and system level heterogeneity: specialized prefill and decode accelerators, broader scale up networking, and decoupled compute-memory enabled by optical I/O. We further hypothesize agent-hardware co design, multiple inference accelerators within one system, and high bandwidth, large capacity memory disaggregation as foundations for adaptation to evolving OI/CF. Together, these directions chart a path to sustain efficiency and capability for large scale agentic AI inference.

Figures (4)

• Figure 1: The traditional roofline model only addresses the leftmost sections (the blue and yellow areas). High MFU indicates the compute-bound region (blue), while high MBU represents the memory bandwidth-bound region (yellow). However, both the roofline model and the concepts of MBU and MFU fail to clearly illustrate under-utilization that arises from memory capacity limitations. We also map actual workloads to these regions, for instance, Decode-FFN (low B, high L) means decode-time FFN block in transformers at low batch size (B) and long sequence length (L).
• Figure 2: Prefill and decode token usage, as well as CF and OI characteristics, for different agents (Chatbot, Coding, Web-use, and Computer-use) using a dense LLaMA3-70B model.
• Figure 3: We compare Multi-head Attention (MHA), Grouped Query Attention (GQA), and Multi-head Latent Attention (MLA) using a 48-layer model with a hidden dimension of 2048 and 32 attention heads, all operating at 16-bit precision.
• Figure 4: Capacity Footprint (for batch sizes 1 and 16) and Decode time OI, comparing dense models and sparse MoE models. The shaded region represents model weights.