Low latency optical-based mode tracking with machine learning deployed on FPGAs on a tokamak

TL;DR

This work tackles the challenge of real-time, microsecond-scale tracking of the $n=1$ MHD mode in tokamaks by deploying a CNN directly on an FPGA embedded in a high-speed frame-grabber used with optical cameras. Through a co-design workflow that includes TensorFlow modeling, hls4ml-based hardware synthesis, quantization, pruning, and careful reuse-factor optimization, the authors achieve a total trigger-to-output latency of $17.6~\mu\mathrm{s}$ and throughput up to $120~\mathrm{kfps}$, with CNN inference taking about $7.7~\mu\mathrm{s}$. The Optimized model, using a $32\times32$ input and aggressive hardware optimizations, meets latency targets while dramatically reducing resource usage, enabling real-time ML-based tokamak diagnostics and prospective control applications on the HBT-EP device. The result demonstrates a scalable end-to-end platform for real-time, ML-driven plasma diagnostics with potential cross-domain utility in other fast-imaging contexts.

Abstract

Active feedback control in magnetic confinement fusion devices is desirable to mitigate plasma instabilities and enable robust operation. Optical high-speed cameras provide a powerful, non-invasive diagnostic and can be suitable for these applications. In this study, we process fast camera data, at rates exceeding 100kfps, on $\textit{in situ}$ Field Programmable Gate Array (FPGA) hardware to track magnetohydrodynamic (MHD) mode evolution and generate control signals in real-time. Our system utilizes a convolutional neural network (CNN) model which predicts the $n$=1 MHD mode amplitude and phase using camera images with better accuracy than other tested non-deep-learning-based methods. By implementing this model directly within the standard FPGA readout hardware of the high-speed camera diagnostic, our mode tracking system achieves a total trigger-to-output latency of 17.6$μ$s and a throughput of up to 120kfps. This study at the High Beta Tokamak-Extended Pulse (HBT-EP) experiment demonstrates an FPGA-based high-speed camera data acquisition and processing system, enabling application in real-time machine-learning-based tokamak diagnostic and control as well as potential applications in other scientific domains.

Figures (11)

• Figure 1: Hardware setup of the high-speed camera diagnostics on HBT-EP as used during the previous run campaign Wei23PPCF, with an actual view of the chamber (similar to Camera 1 view) shown in the top-right inset. The same database was used in this study for training and testing the various CNN models for FPGA implementation as described in the following sections. Reused with permission from Y. Wei et al (2023) PPCF65 074002. Copyright 2023 IOP Publishing.
• Figure 2: Data flow in theproposed camera-and-FPGA-based active feedback control system. The portion of the implementation detailed in this paper is within the solid box. Control signals will be applied to plasmas in upcoming run campaigns.
• Figure 3: Illustration of the firmware design procedures. Results of each step impact choices made during repeated iterations through the process.
• Figure 4: Probability distributions of the amplitude (left) and phase (right) prediction errors over the testing set (ground truth amplitude $>$3G), given by the three candidate CNN models. For each model the results of the Tensorflow implementation are shown in solid lines, and that of the converted HLS implementation are shown in dashed lines. Results of two additional non-deep-learning-based methods using linear regression and SVD-based algorithms Wei23PPCF are included for comparison. All models except the Optimized model use input images of $128\times64$ pixels resolution, whereas the Optimized model uses $32\times32$ pixels resolution. Distributions are obtained using kernel density estimation.
• Figure 5: Comparison of single shot tracking performance of the three candidate models after hls4ml conversion, using testing shot 114467. The 4/1 and 3/1-dominant periods are highlighted in cyan and yellow respectively. Top: Amplitude (left) and phase (right) ground truth calculated from magnetic sensors. Bottom 3 rows: Amplitude (left) and phase (right) prediction errors of the candidate models.