Real-Time Semantic Segmentation on FPGA for Autonomous Vehicles Using LMIINet with the CGRA4ML Framework

TL;DR

This work tackles real-time semantic segmentation for autonomous driving under strict power and latency constraints by mapping a hardware-aware, quantized version of LMIINet onto an FPGA via the CGRA4ML framework. The approach crafts a hardware-friendly encoder–decoder with a compact Flatten Transformer, attends to channel information with simplified CABs, and enables streaming off-chip dataflow to meet real-time requirements. Key contributions include a CGRA4ML-based mapping with operator fusion, a four-stage QAT schedule preserving accuracy at 8-bit precision, and an end-to-end ZCU104 demonstration achieving ~50 ms latency and ~45% mIoU on Cityscapes, along with a discussion of ASIC migration for cost and energy efficiency. The results show that FPGAs, when paired with CGRA4ML and carefully designed lightweight transformers, can deliver competitive segmentation performance at real-time speeds with significantly lower power than GPUs, offering a viable path for autonomous-vehicle perception stacks and edge deployments.

Abstract

Semantic segmentation has emerged as a fundamental problem in computer vision, gaining particular importance in real-time applications such as autonomous driving. The main challenge is achieving high accuracy while operating under computational and hardware constraints. In this research, we present an FPGA-based implementation of real-time semantic segmentation leveraging the lightweight LMIINet architecture and the Coarse-Grained Reconfigurable Array for Machine Learning (CGRA4ML) hardware framework. The model was trained using Quantization-Aware Training (QAT) with 8-bit precision on the Cityscapes dataset, reducing memory footprint by a factor of four while enabling efficient fixed-point computations. Necessary modifications were applied to adapt the model to CGRA4ML constraints, including simplifying skip connections, employing hardware-friendly operations such as depthwise-separable and 1A-1 convolutions, and redesigning parts of the Flatten Transformer. Our implementation achieves approximately 90% pixel accuracy and 45% mean Intersection-over-Union (mIoU), operating in real-time at 20 frames per second (FPS) with 50.1 ms latency on the ZCU104 FPGA board. The results demonstrate the potential of CGRA4ML, with its flexibility in mapping modern layers and off-chip memory utilization for skip connections, provides a path for implementing advanced semantic segmentation networks on FPGA for real-time applications to outperform traditional GPU solutions in terms of power efficiency while maintaining competitive accuracy. The code for this project is publicly available at https://github.com/STAmirr/ cgra4ml_semantic_segmentation

Figures (8)

• Figure 1: LMIINet Architecture ref1
• Figure 2: The diagram of the proposed Lightweight Feature Interaction Bottleneck (LFIB), improved Flatten Transformer, Segmentation Head (SegHead), and Channel Attention Block (CAB). D represents the depth-wise convolution, R is the kernel of dilated convolution, and CS denotes the channel shuffle operation. ref1
• Figure 3: The diagram of the Combination Coefficient learning (CC) scheme, Channel Recurrent Unit (CRU), and the Feature Enhancement (FE) module ref1
• Figure 4: CGRA4ML FPGA Implementation ref4
• Figure 5: Training (orange) and validation (blue) loss curves for quantized LMIINet over 218 epochs. Loss decreases sharply in Phase 1, stabilizes during Phase 2 (with frozen decoder), then drops further in Phase 3 after regularization removal, and finally levels off in Phase 4.