Hardware Acceleration for Neural Networks: A Comprehensive Survey

TL;DR

The paper addresses the bottlenecks in hardware acceleration for neural networks, showing that end-to-end performance is dominated by data movement and memory hierarchy rather than peak arithmetic throughput. It surveys architectures from GPUs to LPUs and in-/near-memory/analog and neuromorphic approaches, and it emphasizes co-design of models, compilers, and hardware to optimize memory bandwidth, interconnects, and KV-cache management for Transformers/LLMs. Key contributions include a unified taxonomy, design patterns (systolic arrays, tiling, dataflow, and memory hierarchies), and a critical discussion of benchmarking and evaluation practices, with open challenges in LLM serving, dynamic sparsity, energy-aware deployment, and reproducible comparisons. The survey highlights that scalable, real-world speedups will come from end-to-end system optimization—balancing compute, memory, and software—rather than raw increases in FLOPs alone, and points toward memory-centric, co-designed architectures for the next generation of neural accelerators.

Abstract

Neural networks have become dominant computational workloads across cloud and edge platforms, but their rapid growth in model size and deployment diversity has exposed hardware bottlenecks increasingly dominated by memory movement, communication, and irregular operators rather than peak arithmetic throughput. This survey reviews the current technology landscape for hardware acceleration of deep learning, spanning GPUs and tensor-core architectures, domain-specific accelerators (TPUs, NPUs), FPGA-based designs, ASIC inference engines, and emerging LLM-serving accelerators such as LPUs, alongside in-/near-memory computing and neuromorphic/analog approaches. We organize the survey using a unified taxonomy across (i) workloads (CNNs, RNNs, GNNs, Transformers/LLMs), (ii) execution settings (training vs.\ inference; datacenter vs.\ edge), and (iii) optimization levers (reduced precision, sparsity and pruning, operator fusion, compilation and scheduling, memory-system/interconnect design). We synthesize key architectural ideas such as systolic arrays, vector and SIMD engines, specialized attention and softmax kernels, quantization-aware datapaths, and high-bandwidth memory, and discuss how software stacks and compilers bridge model semantics to hardware. Finally, we highlight open challenges -- including efficient long-context LLM inference (KV-cache management), robust support for dynamic and sparse workloads, energy- and security-aware deployment, and fair benchmarking -- pointing to promising directions for the next generation of neural acceleration.

Figures (20)

• Figure 1: High-level overview of the hardware acceleration landscape, illustrating the spectrum from general-purpose GPUs to domain-specific TPUs, FPGAs, and ASICs, along with the interaction between compute datapaths, memory hierarchies, and the software stack.
• Figure 2: Key limitations and trade-offs in hardware acceleration design, highlighting the multi-objective optimization problem involving performance (throughput/latency), energy efficiency, and deployment constraints such as area and cost.
• Figure 3: Power consumption analysis highlighting (a) the dominance of data movement energy over arithmetic operations, (b) the impact of reduced precision on energy efficiency, and (c) the trade-offs involved in exploiting sparsity.
• Figure 4: LLM inference bottlenecks and optimization strategies, differentiating between the compute-bound prefill phase and the memory-bandwidth-bound decode phase, and illustrating techniques like KV-cache management, paging, and attention optimization.
• Figure 5: The fundamental trade-off between performance and silicon area/cost, illustrating how increasing parallelism and memory capacity improves throughput but raises manufacturing and packaging costs.