Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A 16 nm 1.60TOPS/W High Utilization DNN Accelerator with 3D Spatial Data Reuse and Efficient Shared Memory Access

Xiaoling Yi, Ryan Antonio, Yunhao Deng, Fanchen Kong, Joren Dumoulin, Jun Yin, Marian Verhelst

TL;DR

Voltra tackles the challenge of achieving high utilization in versatile DNN accelerators by introducing a 3D spatial GEMM core paired with a flexible shared memory and data streamers. The design employs mixed-grained data prefetching, programmable dynamic memory allocation, and time-multiplexing to boost both spatial and temporal utilization, while reducing off-chip communications. Measured on a 16nm chip, Voltra delivers up to 2x spatial utilization gains, 2.12–2.94x temporal utilization improvements, and competitive energy and area efficiency (1.60 TOPS/W, 1.25 TOPS/mm^2) across CNN, RNN, and transformer workloads. Overall, Voltra offers a flexible, high-utilization architecture that competes with state-of-the-art solutions while maintaining low area and energy footprints for diverse workloads.

Abstract

Achieving high compute utilization across a wide range of AI workloads is crucial for the efficiency of versatile DNN accelerators. This paper presents the Voltra chip and its utilization-optimised DNN accelerator architecture, which leverages 3-Dimensional (3D) spatial data reuse along with efficient and flexible shared memory access. The 3D spatial dataflow enables balanced spatial data reuse across three dimensions, improving spatial utilization by up to 2.0x compared to a conventional 2D design. Inside the shared memory access architecture, Voltra incorporates flexible data streamers that enable mixed-grained hardware data pre-fetching and dynamic memory allocation, further improving the temporal utilization by 2.12-2.94x and achieving 1.15-2.36x total latency speedup compared with the non-prefetching and separated memory architecture, respectively. Fabricated in 16nm technology, our chip achieves 1.60 TOPS/W peak system energy efficiency and 1.25 TOPS/mm2 system area efficiency, which is competitive with state-of-the-art solutions while achieving high utilization across diverse workloads.

A 16 nm 1.60TOPS/W High Utilization DNN Accelerator with 3D Spatial Data Reuse and Efficient Shared Memory Access

TL;DR

Voltra tackles the challenge of achieving high utilization in versatile DNN accelerators by introducing a 3D spatial GEMM core paired with a flexible shared memory and data streamers. The design employs mixed-grained data prefetching, programmable dynamic memory allocation, and time-multiplexing to boost both spatial and temporal utilization, while reducing off-chip communications. Measured on a 16nm chip, Voltra delivers up to 2x spatial utilization gains, 2.12–2.94x temporal utilization improvements, and competitive energy and area efficiency (1.60 TOPS/W, 1.25 TOPS/mm^2) across CNN, RNN, and transformer workloads. Overall, Voltra offers a flexible, high-utilization architecture that competes with state-of-the-art solutions while maintaining low area and energy footprints for diverse workloads.

Abstract

Achieving high compute utilization across a wide range of AI workloads is crucial for the efficiency of versatile DNN accelerators. This paper presents the Voltra chip and its utilization-optimised DNN accelerator architecture, which leverages 3-Dimensional (3D) spatial data reuse along with efficient and flexible shared memory access. The 3D spatial dataflow enables balanced spatial data reuse across three dimensions, improving spatial utilization by up to 2.0x compared to a conventional 2D design. Inside the shared memory access architecture, Voltra incorporates flexible data streamers that enable mixed-grained hardware data pre-fetching and dynamic memory allocation, further improving the temporal utilization by 2.12-2.94x and achieving 1.15-2.36x total latency speedup compared with the non-prefetching and separated memory architecture, respectively. Fabricated in 16nm technology, our chip achieves 1.60 TOPS/W peak system energy efficiency and 1.25 TOPS/mm2 system area efficiency, which is competitive with state-of-the-art solutions while achieving high utilization across diverse workloads.
Paper Structure (13 sections, 7 figures, 1 table)

This paper contains 13 sections, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Chip Architecture
  3. 3D Spatial Data Reuse by GEMM Core
  4. Mixed-Grained Data Pre-fetching by Flexible Data Streamers
  5. Programmable Dynamic Memory Allocation
  6. Time-Multiplexing
  7. Auxiliary Modules
  8. Measurement Results
  9. Performance Evaluation
  10. Efficiency Evaluation
  11. State-of-the-Art Comparison
  12. Conclusions
  13. Acknowledgments

Figures (7)

  • Figure 1: (a) 2D spatial array with separated data buffers and dispatchers. (b) 3D spatial array with shared memory and data streamers. (c) Memory usage comparison for the same tiling strategy of ResNet50 between the separated and shared memory.
  • Figure 2: Voltra architecture overview.
  • Figure 3: Flexible data streamer architecture and mixed-grained data prefetching mechanism: (a) fine-grained data prefetching and (b) coarse-grained data prefetching.
  • Figure 4: Dynamic memory allocation example for Multi-Head Attention (MHA). (a) Detailed MHA operation in BERT-Base model with one head and token size=64. (b) Memory allocation for each operand in the MHA computation sequence. (c) Saved memory access count.
  • Figure 5: Chip micrograph and specification.
  • ...and 2 more figures