Segmentation-Free Outcome Prediction from Head and Neck Cancer PET/CT Images: Deep Learning-Based Feature Extraction from Multi-Angle Maximum Intensity Projections (MA-MIPs)

TL;DR

Head and neck cancer prognosis typically depends on ROI segmentation, which is labor-intensive and variable. This work presents a segmentation-free pipeline that automatically crops the head and neck region, derives 72 MA-MIPs from rotated PET volumes, and extracts deep features with pre-trained EfficientNet backbones, which are fused across views and used in a Cox PH model for recurrence-free survival analysis. Evaluated on the HECKTOR2022 dataset (489 patients), the approach achieves a mean $c$-index near 0.72, outperforming the prior HECKTOR2022 winner on training data, and demonstrates improved reproducibility and potential for clinical deployment without manual delineation. The method highlights the value of multi-view PET representations and transfer-learned feature extraction for robust prognostication in head and neck cancer, with implications for scalability and edge-device applicability.

Abstract

We introduce an innovative, simple, effective segmentation-free approach for outcome prediction in head \& neck cancer (HNC) patients. By harnessing deep learning-based feature extraction techniques and multi-angle maximum intensity projections (MA-MIPs) applied to Fluorodeoxyglucose Positron Emission Tomography (FDG-PET) volumes, our proposed method eliminates the need for manual segmentations of regions-of-interest (ROIs) such as primary tumors and involved lymph nodes. Instead, a state-of-the-art object detection model is trained to perform automatic cropping of the head and neck region on the PET volumes. A pre-trained deep convolutional neural network backbone is then utilized to extract deep features from MA-MIPs obtained from 72 multi-angel axial rotations of the cropped PET volumes. These deep features extracted from multiple projection views of the PET volumes are then aggregated and fused, and employed to perform recurrence-free survival analysis on a cohort of 489 HNC patients. The proposed approach outperforms the best performing method on the target dataset for the task of recurrence-free survival analysis. By circumventing the manual delineation of the malignancies on the FDG PET-CT images, our approach eliminates the dependency on subjective interpretations and highly enhances the reproducibility of the proposed survival analysis method.

Figures (4)

• Figure 1: Overview of the proposed method for fully automated segmentation-free outcome prediction for head and neck cancer patients. A) Overall representation of the automated head and neck region detection and cropping. B) Extracting deep features from MA-MIP projections followed by feature fusion step.
• Figure 2: Visual summary of the proposed bounding box definition and cropping for locating the anatomical head and neck region on the PET volumes of the patients based on the coronal and Sagittal MIP CT projections. Locating the bounding boxes on coronal and sagittal CT MIPs based on the AC and SC clavicle joints (left), and the 3D bounding box cropping on the PET volume based on the two 2D bounding boxes on the CT mips (right).
• Figure 3: Box plots of the c-index values over all five test folds. Each plot shows the results using one of the four proposed pooling methods, for EfficientNet large as the feature extraction method, and all six different fusion techniques. Global average pooling, top left. Global max pooling top right. Global median pooling, buttom left, and global standard deviation pooling on the buttom right.
• Figure 4: Box plots summarizing the performance of channel-wise maximum fusion technique(left) and the auto-encoder(right) as the two best fusion methods over all four different global pooling methods, and all five testing folds.