Event-based vision on FPGAs -- a survey
Tomasz Kryjak
TL;DR
This survey addresses the challenge of processing event-based vision data on FPGA and SoC-FPGA platforms, outlining why event cameras offer low latency and high dynamic range but require specialized processing. It surveys two decades of work across filtration, optical flow, stereovision, object detection/tracking, and AI accelerators (SNNs and CNNs), highlighting how researchers adapt FPGA resources to real-time, energy-efficient processing. Key contributions include a consolidated view of 60+ FPGA/SoC FPGA studies (2012–mid-2024), a taxonomy of methods, and critical observations on datasets, benchmarking, and hardware trends. The findings reveal substantial progress in filtering and AI accelerators, but persistent gaps in standardized evaluation, end-to-end AI deployment, and high-resolution full-stack systems, underscoring opportunities for multi-sensor fusion and next-generation FPGA platforms to advance practical, real-time event-based vision. The work advances the field by clarifying the state of FPGA-based event processing, identifying knowledge gaps, and suggesting concrete directions for scalable, energy-efficient neuromorphic vision in robotics and embedded systems.
Abstract
In recent years there has been a growing interest in event cameras, i.e. vision sensors that record changes in illumination independently for each pixel. This type of operation ensures that acquisition is possible in very adverse lighting conditions, both in low light and high dynamic range, and reduces average power consumption. In addition, the independent operation of each pixel results in low latency, which is desirable for robotic solutions. Nowadays, Field Programmable Gate Arrays (FPGAs), along with general-purpose processors (GPPs/CPUs) and programmable graphics processing units (GPUs), are popular architectures for implementing and accelerating computing tasks. In particular, their usefulness in the embedded vision domain has been repeatedly demonstrated over the past 30 years, where they have enabled fast data processing (even in real-time) and energy efficiency. Hence, the combination of event cameras and reconfigurable devices seems to be a good solution, especially in the context of energy-efficient real-time embedded systems. This paper gives an overview of the most important works, where FPGAs have been used in different contexts to process event data. It covers applications in the following areas: filtering, stereovision, optical flow, acceleration of AI-based algorithms (including spiking neural networks) for object classification, detection and tracking, and applications in robotics and inspection systems. Current trends and challenges for such systems are also discussed.