Towards AI Lesion Tracking in PET/CT Imaging: A Siamese-based CNN Pipeline applied on PSMA PET/CT Scans

TL;DR

This paper tackles automated lesion tracking in PSMA-PET/CT to overcome biases from manual index-lesion selection. It introduces a patch-based Siamese CNN pipeline that leverages baseline and follow-up scans after lesion segmentation and affine registration to classify corresponding lesion pairs, exploring both 2D and 3D configurations and multiple patch-types. The best configuration—2D Siamese with single-channel CT patches—achieves $AUC=0.91$ and $accuracy=0.83$, with $89 ext{%}$ re-identification of remaining lesions, demonstrating the approach’s potential to scale analysis to many lesions across patients. The method shows strong promise for improving tumor response assessment and could generalize to other oncologic PET/CT applications, though it relies on accurate segmentation and may benefit from future architectural enhancements such as transformers.

Abstract

Assessing tumor response to systemic therapies is one of the main applications of PET/CT. Routinely, only a small subset of index lesions out of multiple lesions is analyzed. However, this operator dependent selection may bias the results due to possible significant inter-metastatic heterogeneity of response to therapy. Automated, AI based approaches for lesion tracking hold promise in enabling the analysis of many more lesions and thus providing a better assessment of tumor response. This work introduces a Siamese CNN approach for lesion tracking between PET/CT scans. Our approach is applied on the laborious task of tracking a high number of bone lesions in full-body baseline and follow-up [68Ga]Ga- or [18F]F-PSMA PET/CT scans after two cycles of [177Lu]Lu-PSMA therapy of metastatic castration resistant prostate cancer patients. Data preparation includes lesion segmentation and affine registration. Our algorithm extracts suitable lesion patches and forwards them into a Siamese CNN trained to classify the lesion patch pairs as corresponding or non-corresponding lesions. Experiments have been performed with different input patch types and a Siamese network in 2D and 3D. The CNN model successfully learned to classify lesion assignments, reaching a lesion tracking accuracy of 83 % in its best configuration with an AUC = 0.91. For remaining lesions the pipeline accomplished a re-identification rate of 89 %. We proved that a CNN may facilitate the tracking of multiple lesions in PSMA PET/CT scans. Future clinical studies are necessary if this improves the prediction of the outcome of therapies.

Figures (9)

• Figure 1: PSMA-PET/CT Scans showing a Male 73 year old Patient with Partial Response to [$^{177}$Lu]Lu-PSMA Therapy: The baseline scan prior to therapy is shown on the left side, the follow-up scan after two therapy cycles on the right side: A&H Maximum intensity projections (MIP), B&E axial CT slices of the upper thorax, C&F corresponding axial PET slices, D&G PET/CT overlay. The MIP show a clear reduction of tumor burdon with less bone lesions. Physiologic PSMA-uptake can be found in the lacrimal glands, salivatory glands, liver, spleen, small bowel, kidneys and bladder. The white arrows indicate resolved lesions, the red arrows remaining lesions. A complete comparative 3D analysis of all lesions is laborious and could be facilitated by an automated tracking algorithm.
• Figure 2: Processing workflow: Schematic presentation of the proposed pipeline for AI-based PET/CT lesion tracking. Data preparation of the baseline and follow-up scans include a binary bone lesion segmentation (Seg.) and an affine image registration. Patches are extracted around every lesion and then analyzed by a siamese CNN, which assigns the lesions in the baseline and follow-up PET/CT scans.
• Figure 3: 2D lesion patch extraction: An axial 50$\times$50 pixel patch is cropped around the determined extraction point $p(x,y,z)$.
• Figure 4: Principle of the patch extraction algorithm: (A) Unsuitable patch pair after extraction at center of mass: A large baseline lesion shrank and divided into three smaller lesions in the follow up-scan. The example shows a baseline and follow-up lesion patch of the left scapula extracted at the lesions' center of mass (CoM). Patch extraction at each lesion's CoM can lead to patch pairs showing different anatomical environments, even though they show corresponding lesions. For this reason, a detailed patch extraction algorithm determines the suitable patch extraction point $p(x,y,z)$. (B) Patch extraction cases: In the algorithm a case distinction hierarchically applies the four cases A-D to find suitable points $p(x,y,z)$ within the baseline and follow-up lesion for the patch pair extraction. If one case does not lead to a result, the algorithm passes on to the next one. For each case, the illustration shows the baseline lesion(s) on the left side and the follow-up lesion(s) on the right side. The gray cross-sectional areas indicate the final extracted axial patches. $\mathbf{T}$ and $\mathbf{T^{-1}}$ indicate point transfers to the respective other scan (eq. \ref{['eq:TransferBaslinetoFollowUp']}, eq. \ref{['eq:TransferFollow-UptoBaseline']}).
• Figure 5: Structure of the siamese network: Patches are processed by two parallel CNN branches with shared weights, whose output is merged by a $\mathbf{L_1}$-layer. Architecture details of the branches are shown in table \ref{['tab:SiameseNetwork']}.