CaMDN: Enhancing Cache Efficiency for Multi-tenant DNNs on Integrated NPUs

TL;DR

CaMDN tackles cache inefficiency in multi-tenant DNNs on integrated NPUs by co-designing cache architecture and scheduling. It introduces an NPU-controlled cache with model-exclusive regions, a cache-aware mapping mechanism with a heuristic-solver-hybrid mapper and layer-block data reuse, and a dynamic cache allocation algorithm to adapt at runtime. The approach reduces memory accesses by $33.4\%$ and achieves up to $2.56\times$ model speedup (average $1.88\times$). Evaluations on a cycle-accurate simulator with eight DNNs show substantial latency, memory-access, and QoS improvements with modest area overhead. This work enables more efficient multi-tenant DNN deployment on integrated NPUs, with practical impact for edge and server SoCs.

Abstract

With the rapid development of DNN applications, multi-tenant execution, where multiple DNNs are co-located on a single SoC, is becoming a prevailing trend. Although many methods are proposed in prior works to improve multi-tenant performance, the impact of shared cache is not well studied. This paper proposes CaMDN, an architecture-scheduling co-design to enhance cache efficiency for multi-tenant DNNs on integrated NPUs. Specifically, a lightweight architecture is proposed to support model-exclusive, NPU-controlled regions inside shared cache to eliminate unexpected cache contention. Moreover, a cache scheduling method is proposed to improve shared cache utilization. In particular, it includes a cache-aware mapping method for adaptability to the varying available cache capacity and a dynamic allocation algorithm to adjust the usage among co-located DNNs at runtime. Compared to prior works, CaMDN reduces the memory access by 33.4% on average and achieves a model speedup of up to 2.56$\times$ (1.88$\times$ on average).

Figures (9)

• Figure 1: An overview of architecture and scheduling for multi-tenant DNN execution on integrated NPUs. Cache contention occurs between co-located DNNs.
• Figure 2: Cache hit rate, memory access and average latency with different settings on number of DNNs and cache capacity.
• Figure 3: Percentages of data with different reuse counts and reuse distances on shared cache in benchmark DNN models.
• Figure 4: The overview of CaMDN architecture. CaMDN installs an NEC in each shared cache slice and a CPT in each NPU.
• Figure 5: Implementation details of CaMDN architecture. (a) NEC handles NPU-specific requests. (b) CPT translates vcaddr into pcaddr that indexes the targeted cache line in data arrays.