Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AP-DRL: A Synergistic Algorithm-Hardware Framework for Automatic Task Partitioning of Deep Reinforcement Learning on Versal ACAP

Enlai Li, Zhe Lin, Sharad Sinha, Wei Zhang

Abstract

Deep reinforcement learning has demonstrated remarkable success across various domains. However, the tight coupling between training and inference processes makes accelerating DRL training an essential challenge for DRL optimization. Two key issues hinder efficient DRL training: (1) the significant variation in computational intensity across different DRL algorithms and even among operations within the same algorithm complicates hardware platform selection, while (2) DRL's wide dynamic range could lead to substantial reward errors with conventional FP16+FP32 mixed-precision quantization. While existing work has primarily focused on accelerating DRL for specific computing units or optimizing inference-stage quantization, we propose AP-DRL to address the above challenges. AP-DRL is an automatic task partitioning framework that harnesses the heterogeneous architecture of AMD Versal ACAP (integrating CPUs, FPGAs, and AI Engines) to accelerate DRL training through intelligent hardware-aware optimization. Our approach begins with bottleneck analysis of CPU, FPGA, and AIE performance across diverse DRL workloads, informing the design principles for AP-DRL's inter-component task partitioning and quantization optimization. The framework then addresses the challenge of platform selection through design space exploration-based profiling and ILP-based partitioning models that match operations to optimal computing units based on their computational characteristics. For the quantization challenge, AP-DRL employs a hardware-aware algorithm coordinating FP32 (CPU), FP16 (FPGA/DSP), and BF16 (AI Engine) operations by leveraging Versal ACAP's native support for these precision formats. Comprehensive experiments indicate that AP-DRL can achieve speedup of up to 4.17$\times$ over programmable logic and up to 3.82$\times$ over AI Engine baselines while maintaining training convergence.

AP-DRL: A Synergistic Algorithm-Hardware Framework for Automatic Task Partitioning of Deep Reinforcement Learning on Versal ACAP

Abstract

Deep reinforcement learning has demonstrated remarkable success across various domains. However, the tight coupling between training and inference processes makes accelerating DRL training an essential challenge for DRL optimization. Two key issues hinder efficient DRL training: (1) the significant variation in computational intensity across different DRL algorithms and even among operations within the same algorithm complicates hardware platform selection, while (2) DRL's wide dynamic range could lead to substantial reward errors with conventional FP16+FP32 mixed-precision quantization. While existing work has primarily focused on accelerating DRL for specific computing units or optimizing inference-stage quantization, we propose AP-DRL to address the above challenges. AP-DRL is an automatic task partitioning framework that harnesses the heterogeneous architecture of AMD Versal ACAP (integrating CPUs, FPGAs, and AI Engines) to accelerate DRL training through intelligent hardware-aware optimization. Our approach begins with bottleneck analysis of CPU, FPGA, and AIE performance across diverse DRL workloads, informing the design principles for AP-DRL's inter-component task partitioning and quantization optimization. The framework then addresses the challenge of platform selection through design space exploration-based profiling and ILP-based partitioning models that match operations to optimal computing units based on their computational characteristics. For the quantization challenge, AP-DRL employs a hardware-aware algorithm coordinating FP32 (CPU), FP16 (FPGA/DSP), and BF16 (AI Engine) operations by leveraging Versal ACAP's native support for these precision formats. Comprehensive experiments indicate that AP-DRL can achieve speedup of up to 4.17 over programmable logic and up to 3.82 over AI Engine baselines while maintaining training convergence.

Paper Structure

This paper contains 20 sections, 2 equations, 15 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Deep Reinforcement Learning Algorithm
  4. The Versal ACAP Architecture
  5. Quantization in DRL
  6. Bottleneck Analysis
  7. Performance Analysis of PS, PL, and AIE
  8. Design Principles
  9. The AP-DRL Framework
  10. Workflow Overview of AP-DRL
  11. Partitioning Granularity and Profiling
  12. Partitioning Algorithm
  13. Hardware-Aware Quantization Algorithm
  14. Data Precision Analysis
  15. Quantization Algorithm
  16. ...and 5 more sections

Figures (15)

  • Figure 1: Workflow overview of the DRL algorithm.
  • Figure 2: The overview of Versal ACAP (AI Edge series) architecture.
  • Figure 3: The binary format of BF16, FP16 and FP32.
  • Figure 4: Log-normalized execution time of three DRL algorithm-environment combinations for single training timestep on PS, PL, and AIE-ML processors on the VEK280 platform (hardware emulation), with performance profiled by the Vitis Analyzer. Detailed configurations of these three combinations can be found in Table \ref{['table_benchmark_config']}.
  • Figure 5: Execution time breakdown of one single training timestep on VEK280 PS (hardware emulation), with performance profiled through the xiltimer APIs. The configuration details of the three DRL algorithm-environment combinations can be found in Table \ref{['table_benchmark_config']}.
  • ...and 10 more figures