A Reconfigurable Framework for AI-FPGA Agent Integration and Acceleration

TL;DR

The paper tackles the challenge of deploying real-time, energy-efficient AI inference in resource-constrained environments by introducing AI-FPGA Agent, a co-design framework that combines a runtime CPU-side agent with a parameterizable FPGA accelerator core. The CPU agent dynamically partitions neural networks, schedules FPGA offloads for compute-intensive layers, and orchestrates data transfers, while the FPGA core implements dataflow, quantized operations, and layer-specific pipelines. Key contributions include a modular software-hardware flow, tile-based data orchestration with double buffering, and demonstration on a Xilinx KV260 platform showing over a 10× reduction in latency compared to CPU baselines and 2–3× higher energy efficiency than a mid-range GPU, with accuracy within 0.2% of full-precision references. The work highlights the practicality of agent-driven FPGA acceleration for low-latency AI and sets the stage for extending to transformer architectures, partial reconfiguration, and tighter integration with ML frameworks, enabling scalable and energy-efficient AI deployment across diverse domains.

Abstract

Artificial intelligence (AI) is increasingly deployed in real-time and energy-constrained environments, driving demand for hardware platforms that can deliver high performance and power efficiency. While central processing units (CPUs) and graphics processing units (GPUs) have traditionally served as the primary inference engines, their general-purpose nature often leads to inefficiencies under strict latency or power budgets. Field-Programmable Gate Arrays (FPGAs) offer a promising alternative by enabling custom-tailored parallelism and hardware-level optimizations. However, mapping AI workloads to FPGAs remains challenging due to the complexity of hardware-software co-design and data orchestration. This paper presents AI FPGA Agent, an agent-driven framework that simplifies the integration and acceleration of deep neural network inference on FPGAs. The proposed system employs a runtime software agent that dynamically partitions AI models, schedules compute-intensive layers for hardware offload, and manages data transfers with minimal developer intervention. The hardware component includes a parameterizable accelerator core optimized for high-throughput inference using quantized arithmetic. Experimental results demonstrate that the AI FPGA Agent achieves over 10x latency reduction compared to CPU baselines and 2-3x higher energy efficiency than GPU implementations, all while preserving classification accuracy within 0.2% of full-precision references. These findings underscore the potential of AI-FPGA co-design for scalable, energy-efficient AI deployment.

Figures (4)

• Figure 1: Adapted from zheng2024 Overview of the Q-learning-enhanced scheduling agent. The agent observes the environment's state and reward signals, updates Q-values using temporal difference learning, and selects actions (FPGA offload) via an $\varepsilon$-greedy policy. Synchronization between the primary Q-table $Q_A$ and target Q-table $Q_B$ stabilizes learning.
• Figure 2: Overview of the AI_FPGA_Agent software-to-hardware flow. The application layer interfaces with the AI_FPGA_Agent framework, which communicates with a SystemC-based simulation stack and a hardware accelerator driver. System-level verification is performed through a SystemC testbench and behavioral model before synthesizing the final FPGA bitstream and deploying the hardware accelerator.
• Figure 3: System-level architecture of the proposed LLM inference pipeline on the Xilinx KV260 platform. The model weights and KV cache reside in external DDR4 memory, with 85% bandwidth utilization during inference. A bare-metal host application orchestrates tokenization and decoding via the PS CPU, while quantized model layers (LLaMA2-7B AWQ-4bit) are accelerated in programmable logic using dedicated compute modules such as RoPE, RMSNorm, Softmax, and SiLU. The inference engine communicates over a 64-bit AXI bus at 2400 Mbps and uses DMA to stream data between the PL and memory.
• Figure 4: Adapted from patra2024 LLM-guided hardware design and verification workflow. The process begins with a functional specification and progresses through synthesis, simulation, timing analysis, and layout generation, integrating multiple checkpoints where large language models (LLMs) provide feedback, generate Verilog, and assist in constraint tuning. Reflection prompts help iteratively refine the design through self-assessment and correction.