Benchmarking Deep Learning-Based Reconstruction Methods for Photoacoustic Computed Tomography with Clinically Relevant Synthetic Datasets

TL;DR

This work addresses the lack of standardized, clinically meaningful evaluation for deep learning in photoacoustic tomography by introducing an open benchmarking framework that combines anatomically plausible synthetic breast phantoms with high-fidelity forward models and both traditional and task-based IQ metrics. It provides three progressively realistic 2D datasets (over 11k samples each) and a two-tier evaluation protocol based on relative MSE/SSIM as well as binary detection and detection-localization tasks evaluated with CHO observers, reporting AUC and ALROC. A benchmarking study comparing DL-based data- and image-domain methods against physics-based reconstructions reveals that DL methods can achieve favorable traditional IQ but often fail to detect or localize clinically significant, deep, low-contrast lesions, whereas physics-based and hybrid approaches may offer superior task-driven performance. The framework enables reproducible, objective comparisons and supports method development and system optimization, with public data and evaluation code to foster broader adoption and eventual clinical translation; a 3D extension and expanded modeling are planned to further enhance realism and generalizability. $p_0(oldsymbol{r}) = \Gamma(oldsymbol{r}) A(oldsymbol{r})$ ground truth and task-based IQ metrics are central to aligning DL reconstructions with diagnostic utility.

Abstract

Deep learning (DL)-based image reconstruction methods for photoacoustic computed tomography (PACT) have developed rapidly in recent years. However, most existing methods have not employed standardized datasets, and their evaluations rely on traditional image quality (IQ) metrics that may lack clinical relevance. The absence of a standardized framework for clinically meaningful IQ assessment hinders fair comparison and raises concerns about the reproducibility and reliability of reported advancements in PACT. A benchmarking framework is proposed that provides open-source, anatomically plausible synthetic datasets and evaluation strategies for DL-based acoustic inversion methods in PACT. The datasets each include over 11,000 two-dimensional (2D) stochastic breast objects with clinically relevant lesions and paired measurements at varying modeling complexity. The evaluation strategies incorporate both traditional and task-based IQ measures to assess fidelity and clinical utility. A preliminary benchmarking study is conducted to demonstrate the framework's utility by comparing DL-based and physics-based reconstruction methods. The benchmarking study demonstrated that the proposed framework enabled comprehensive, quantitative comparisons of reconstruction performance and revealed important limitations in certain DL-based methods. Although they performed well according to traditional IQ measures, they often failed to accurately recover lesions. This highlights the inadequacy of traditional metrics and motivates the need for task-based assessments. The proposed benchmarking framework enables systematic comparisons of DL-based acoustic inversion methods for 2D PACT. By integrating clinically relevant synthetic datasets with rigorous evaluation protocols, it enables reproducible, objective assessments and facilitates method development and system optimization in PACT.

Figures (10)

• Figure 1: 3D breast tissue label map with a numerical lesion model inserted in four predetermined locations. Lesions at Locations 1 and 2 are positioned at a depth of 10 mm, and those at Locations 3 and 4 at 20 mm, representing shallow and deep cases for assessing light fluence attenuation with depth. Polar angles ($\theta$) of 100$^\circ$ and 140$^\circ$ correspond to regions near the chest wall and nipple, respectively, enabling evaluation under distinct illumination conditions, while azimuthal angles ($\phi$) are separated to minimize optical interference within the same NBP.
• Figure 2: 2D object generation process: (a) A 3D tissue label map of a stochastic NBP is used to assign optical and acoustic properties. (b) A 3D initial pressure map is generated via photon transport simulation based on the optical property distributions. (c) A thin slab is extracted from the 3D initial pressure map. (d) The slab is averaged along the z-axis to produce a 2D initial pressure map.
• Figure 3: Representative 2D lesion-present object examples from four breast density types. From left to right: initial pressure distribution, SOS, acoustic density map, and attenuation coefficient map. Lesion region is zoomed in. From up to bottom: (a) almost entirely fatty, (b) scattered areas of fibroglandular density, (c) heterogeneously dense, and (d) extremely dense.
• Figure 4: Virtual PACT measurement acquisition designs for the three datasets. Three datasets represent increasing modeling complexity, from an acoustically homogeneous medium with point-like transducers to an acoustically heterogeneous medium with finite-size line transducers.
• Figure 5: ROI image pairs used for task-based IQ assessment. The left column shows the ROI extraction locations (circles) on 3D numerical breast phantoms for each task. The right panel displays the corresponding ROI images extracted from the true object. For the localization task, each group (cyan boxes) consists of one lesion-present ROI image at a specific location and three lesion-absent images from the other locations. For the detection task, each pair (green boxes) consists of a lesion-present ROI image and a lesion-absent ROI image from the same anatomical location.