Addressing memory bandwidth scalability in vector processors for streaming applications

TL;DR

The paper investigates memory bandwidth as a key bottleneck in data-parallel AI/ML workloads and proposes Provet, a vector-architecture extension with a three-tier on-chip memory hierarchy (local, intermediate, global), an ultra-wide shallow SRAM, a Very Wide Register, and dedicated data shufflers to decouple bandwidth from data reuse. It provides a detailed architectural description, including components such as the VWR, tile and VFU shufflers, and a distributed control scheme, and demonstrates CNN mapping capabilities with template-based libraries. Through experimental comparisons against SAs (Eyeriss, TPU) and a vector/GPU baseline (ARA, Nvidia Ampere), Provet achieves higher PE utilization and lower global-memory accesses and latency, particularly when data reuse is limited (e.g., MobileNet). The findings suggest that a wide, shallow on-chip memory with flexible data movement can scale bandwidth with the number of PEs, offering practical benefits for real-world CNN workloads and providing a foundation for software libraries that simplify mapping to such architectures. The approach highlights the practical impact of rethinking memory hierarchy design to sustain performance and energy efficiency in next-generation AI accelerators.

Abstract

As the size of artificial intelligence and machine learning (AI/ML) models and datasets grows, the memory bandwidth becomes a critical bottleneck. The paper presents a novel extended memory hierarchy that addresses some major memory bandwidth challenges in data-parallel AI/ML applications. While data-parallel architectures like GPUs and neural network accelerators have improved power performance compared to traditional CPUs, they can still be significantly bottlenecked by their memory bandwidth, especially when the data reuse in the loop kernels is limited. Systolic arrays (SAs) and GPUs attempt to mitigate the memory bandwidth bottleneck but can still become memory bandwidth throttled when the amount of data reuse is not sufficient to confine data access mostly to the local memories near to the processing. To mitigate this, the proposed architecture introduces three levels of on-chip memory -- local, intermediate, and global -- with an ultra-wide register and data-shufflers to improve versatility and adaptivity to varying data-parallel applications. The paper explains the innovations at a conceptual level and presents a detailed description of the architecture innovations. We also map a representative data-parallel application, like a convolutional neural network (CNN), to the proposed architecture and quantify the benefits vis-a-vis GPUs and repersentative accelerators based on systolic arrays and vector processors.

Figures (11)

• Figure 1: Comparison of the memory hierarchies. The numbered white arrows show the possible paths data can take in the architecture when moving up and down the memory hierarchy through the processing elements (PEs)
• Figure 2: Simplified SRAM memory structure and energy cost estimations.
• Figure 3: Memory hierarchy and interconnect of two implementation variants of the proposed architecture.
• Figure 4: Proposed architecture overview. The ultra-wide elements (4096 bits) and the SIMD unit (512 bits) are not drawn to scale to simplify the diagram.
• Figure 5: Comparison between Provet and an ideal systolic array