Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Instruction-Based Coordination of Heterogeneous Processing Units for Acceleration of DNN Inference

Anastasios Petropoulos, Theodore Antonakopoulos

TL;DR

The paper tackles the challenge of accelerating DNN inference on FPGA by coordinating multiple heterogeneous PUs without reconfiguring hardware. It introduces an instruction-based coordination paradigm built around an Instruction Controller Unit (ICU) per PU and a peer-to-peer Instruction Synchronization Unit (ISU), enabling flexible pipeline and hybrid parallelism. A comprehensive DNN compilation framework translates models into executable instruction sequences, supporting topology-contiguous partitioning, memory-aware scheduling, and runtime deployment across diverse PU configurations. Experimental validation on ResNet-50 demonstrates substantial gains in compute efficiency and throughput, while enabling runtime adaptation between single-batch pipelining and multi-batch processing, thereby offering design space exploration and deployment flexibility for FPGA-based DNN accelerators.

Abstract

This paper presents an instruction-based coordination architecture for Field-Programmable Gate Array (FPGA)-based systems with multiple high-performance Processing Units (PUs) for accelerating Deep Neural Network (DNN) inference. This architecture enables programmable multi-PU synchronization through instruction controller units coupled with peer-to-peer instruction synchronization units, utilizing instruction types organized into load, compute, and store functional groups. A compilation framework is presented that transforms DNN models into executable instruction programs, enabling flexible partitioning of DNN models into topologically contiguous subgraphs mapped to available PUs. Multiple deployment strategies are supported, enabling pipeline parallelism among PUs and batch-level parallelism across different PU subsets, with runtime switching among them without FPGA reconfiguration. The proposed approach enables design space exploration, supporting dynamic trade-offs between single-batch and multi-batch performance. Experimental results on ResNet-50 demonstrate notable compute efficiency, up to $98\%$, and throughput efficiency gains, up to $2.7\times$, over prior works across different configurations.

Instruction-Based Coordination of Heterogeneous Processing Units for Acceleration of DNN Inference

TL;DR

The paper tackles the challenge of accelerating DNN inference on FPGA by coordinating multiple heterogeneous PUs without reconfiguring hardware. It introduces an instruction-based coordination paradigm built around an Instruction Controller Unit (ICU) per PU and a peer-to-peer Instruction Synchronization Unit (ISU), enabling flexible pipeline and hybrid parallelism. A comprehensive DNN compilation framework translates models into executable instruction sequences, supporting topology-contiguous partitioning, memory-aware scheduling, and runtime deployment across diverse PU configurations. Experimental validation on ResNet-50 demonstrates substantial gains in compute efficiency and throughput, while enabling runtime adaptation between single-batch pipelining and multi-batch processing, thereby offering design space exploration and deployment flexibility for FPGA-based DNN accelerators.

Abstract

This paper presents an instruction-based coordination architecture for Field-Programmable Gate Array (FPGA)-based systems with multiple high-performance Processing Units (PUs) for accelerating Deep Neural Network (DNN) inference. This architecture enables programmable multi-PU synchronization through instruction controller units coupled with peer-to-peer instruction synchronization units, utilizing instruction types organized into load, compute, and store functional groups. A compilation framework is presented that transforms DNN models into executable instruction programs, enabling flexible partitioning of DNN models into topologically contiguous subgraphs mapped to available PUs. Multiple deployment strategies are supported, enabling pipeline parallelism among PUs and batch-level parallelism across different PU subsets, with runtime switching among them without FPGA reconfiguration. The proposed approach enables design space exploration, supporting dynamic trade-offs between single-batch and multi-batch performance. Experimental results on ResNet-50 demonstrate notable compute efficiency, up to , and throughput efficiency gains, up to , over prior works across different configurations.

Paper Structure

This paper contains 17 sections, 6 figures, 3 tables.

Table of Contents

  1. Introduction and Related Work
  2. Hardware Foundation
  3. Baseline Architecture
  4. Instruction Set Architecture
  5. Multi-PU Coordination Architecture
  6. Instruction Synchronization Network
  7. Instruction Controller Unit
  8. Two-PU Pipeline Example
  9. DNN Compilation Framework
  10. DNN Model Processing and Profiling
  11. Partitioning and Scheduling
  12. Pipeline Memory Optimization
  13. Instruction Generation
  14. Implementation and Evaluation
  15. Performance Analysis
  16. ...and 2 more sections

Figures (6)

  • Figure 1: System architecture of multiple PUs. Inset: PU architecture petropoulos2025.
  • Figure 2: Instruction-based coordination architecture: (a) System architecture with PUs distribution. (b) ISU. (c) Control token PU-to-PU latencies. (d) ICU.
  • Figure 3: Two-PU pipeline coordination example: instructions/timing overview.
  • Figure 4: DNN Compilation Framework: (a) DNN model preparation. (b) Node fusion and parser. (c) Node profiler. (d) DNN graph partitioning to PUs and weight transfer scheduling. (e) Tensors buffer optimization and liveness analysis. (f) Instruction generation.
  • Figure 5: Three-step DSE methodology.
  • ...and 1 more figures