DCO: Dynamic Cache Orchestration for LLM Accelerators through Predictive Management

TL;DR

This work addresses memory bottlenecks in multi-core AI accelerators by replacing deep scratchpad hierarchies with a shared Last-Level Cache (LLC) and introducing a Tensor Management Unit (TMU) to expose tensor-level metadata. The authors propose Dynamic Cache Orchestration (DCO), integrating dead-block prediction, anti-thrashing, and dynamic bypassing to maximize data reuse under large working sets, with coordinated hardware-software metadata. They validate the approach via cycle-level simulation and develop a scalable analytical model to extend findings to larger workloads, achieving up to 1.8x speedups and demonstrating robustness across varied dataflows and models; RTL synthesis shows practical area and power feasibility at 15 nm. The results highlight the practical viability of shared-cache architectures for future AI accelerators and offer a disciplined framework for memory-subsystem optimization in Transformers and LLM workloads.

Abstract

The rapid adoption of large language models (LLMs) is pushing AI accelerators toward increasingly powerful and specialized designs. Instead of further complicating software development with deeply hierarchical scratchpad memories (SPMs) and their asynchronous management, we investigate the opposite point of the design spectrum: a multi-core AI accelerator equipped with a shared system-level cache and application-aware management policies, which keeps the programming effort modest. Our approach exploits dataflow information available in the software stack to guide cache replacement (including dead-block prediction), in concert with bypass decisions and mechanisms that alleviate cache thrashing. We assess the proposal using a cycle-accurate simulator and observe substantial performance gains (up to 1.80x speedup) compared with conventional cache architectures. In addition, we build and validate an analytical model that takes into account the actual overlapping behaviors to extend the measurement results of our policies to real-world larger-scale workloads. Experiment results show that when functioning together, our bypassing and thrashing mitigation strategies can handle scenarios both with and without inter-core data sharing and achieve remarkable speedups. Finally, we implement the design in RTL and the area of our design is $\mathbf{0.064mm^2}$ with 15nm process, which can run at 2 GHz clock frequency. Our findings explore the potential of the shared cache design to assist the development of future AI accelerator systems.

Figures (10)

• Figure 1: (a) A typical AI accelerator structure. (b) SoC with accelerators directly connected to DRAM via DMA. (c) SoC with accelerator cores connected to a shared SPM. (d) SoC with accelerator cores connected to the CPU L2 cache. (e) (ours) SoC with accelerator cores first connected to a shared cache (assisted by an additional "TMU").
• Figure 2: (a) Pseudo code of tiled MatMul. nAcc: expected number of accesses of each tile in this matrix; getTile: load the tile into the AI core private SPM; setTile: store the tile into the memory system from the AI core private SPM. Lines in red are manually-inserted registration codes to pass dataflow-related information to TMU. (b) System architecture and workflow with TMU. The interaction between LLC and TMU is elaborated in Section \ref{['DBPDetail']}.
• Figure 3: Cache hit rate over time for LRU and our policies for a 4MB LLC (Gemma3-27B, 2K sequence length). The at policy demonstrates a consistently higher hit rate compared to LRU under contended conditions.
• Figure 4: Execution time for different LLC replacement policies and capacities. at: anti-thrashing. Unless otherwise specified, bypass for spatial group allocation cases refers to the gqa_bypass variant.
• Figure 5: Execution time for different anti-thrashing configurations (B_BITS) and capacities. Lower values indicate better performance (Gemma3-27B, 4K sequence length, temporal group allocation).