Unlocking Real-Time Fluorescence Lifetime Imaging: Multi-Pixel Parallelism for FPGA-Accelerated Processing

TL;DR

This work implemented a GRU-based sequence-to-sequence (Seq2Seq) model on an FPGA board compatible with time-resolved cameras, integrating a GRU-based Seq2Seq model and its compressed version, called Seq2SeqLite, generated through knowledge distillation, to process multiple pixels in parallel, reducing latency compared to sequential processing.

Abstract

Fluorescence lifetime imaging (FLI) is a widely used technique in the biomedical field for measuring the decay times of fluorescent molecules, providing insights into metabolic states, protein interactions, and ligand-receptor bindings. However, its broader application in fast biological processes, such as dynamic activity monitoring, and clinical use, such as in guided surgery, is limited by long data acquisition times and computationally demanding data processing. While deep learning has reduced post-processing times, time-resolved data acquisition remains a bottleneck for real-time applications. To address this, we propose a method to achieve real-time FLI using an FPGA-based hardware accelerator. Specifically, we implemented a GRU-based sequence-to-sequence (Seq2Seq) model on an FPGA board compatible with time-resolved cameras. The GRU model balances accurate processing with the resource constraints of FPGAs, which have limited DSP units and BRAM. The limited memory and computational resources on the FPGA require efficient scheduling of operations and memory allocation to deploy deep learning models for low-latency applications. We address these challenges by using STOMP, a queue-based discrete-event simulator that automates and optimizes task scheduling and memory management on hardware. By integrating a GRU-based Seq2Seq model and its compressed version, called Seq2SeqLite, generated through knowledge distillation, we were able to process multiple pixels in parallel, reducing latency compared to sequential processing. We explore various levels of parallelism to achieve an optimal balance between performance and resource utilization. Our results indicate that the proposed techniques achieved a 17.7x and 52.0x speedup over manual scheduling for the Seq2Seq model and the Seq2SeqLite model, respectively.

Figures (6)

• Figure 1: (Left) Deep Sequence-to-Sequence (Seq2Seq) model featuring two GRU cells in both the encoder and decoder, with 128 hidden units in the first GRU cell of the encoder. The encoder processes the input time sequence and passes its final hidden states to initialize the decoder, which updates its hidden states to generate the output. (Right) Seq2SeqLite model, a compressed version of the deep Seq2Seq model, generated using quantization-aware training (QAT) combined with knowledge distillation (KD).
• Figure 2: (Top) Time-series data points captured using a large-format time-resolved camera in Macroscopic Fluorescence Lifetime Imaging (MFLI). These experimental, wide-field, time-resolved images illustrate the entire body of a nude mouse containing HER2+ tumor xenografts labeled with Alexa Fluor 700 conjugated to Trastuzumab for a non-invasive drug-target binding MFLI study. (Bottom) Schematic representation of parallel data processing for multiple pixels with different temporal point spread functions (TPSFs)
• Figure 3: Utilization of DSP 0 of 128 when executing the encoder layer on one pixel vs. two pixels in parallel.
• Figure 4: Parallel execution results for Seq2Seq (left) and Seq2SeqLite (right) models. The top row shows the number of pixels processed in parallel for the encoder, and the bottom row illustrates the same for the decoder. Both execution time and speedup over manual processing are displayed for different parallel pixel configurations.
• Figure 5: (a) Parallel execution results for the post processing step. Both execution time and speedup over manual processing are displayed for different parallel pixel configurations (b) Speedup comparison between Seq2Seq and Seq2SeqLite models. The bars represent the relative speedup for three configurations: Seq2Seq with 64 pixels, Seq2SeqLite (manual), and Seq2SeqLite with 256 pixels. Results are shown separately for the encoder and decoder stages.