Image segmentation of treated and untreated tumor spheroids by Fully Convolutional Networks

TL;DR

This work tackles the challenge of segmenting treated and untreated tumor spheroids in brightfield 3D cultures to enable high-throughput analysis of therapy response. It develops automatic segmentation using two FCN architectures, UNet and HRNet, and performs a systematic hyperparameter optimization (backbone, transfer learning, augmentation, resolution, loss, optimizer) to achieve high accuracy even when spheroids are obscured by debris after radiotherapy. The optimized UNet, with ResNet34, half-resolution inputs, Dice loss, and RAdam-Lookahead, outperforms HRNet on independent data, delivering IoU-like overlap close to the expert agreement and a low average radial error, while maintaining fast GPU inference and practical applicability. The approach is validated on large independent datasets across two HNSCC cell lines and various treatment conditions, and it is complemented by a publicly available tool and data to facilitate automated extraction of spheroid metrics and potential future tumor-spheroid fate forecasting in preclinical radiobiology.

Abstract

Multicellular tumor spheroids (MCTS) are advanced cell culture systems for assessing the impact of combinatorial radio(chemo)therapy. They exhibit therapeutically relevant in-vivo-like characteristics from 3D cell-cell and cell-matrix interactions to radial pathophysiological gradients related to proliferative activity and nutrient/oxygen supply, altering cellular radioresponse. State-of-the-art assays quantify long-term curative endpoints based on collected brightfield image time series from large treated spheroid populations per irradiation dose and treatment arm. Here, spheroid control probabilities are documented analogous to in-vivo tumor control probabilities based on Kaplan-Meier curves. This analyses require laborious spheroid segmentation of up to 100.000 images per treatment arm to extract relevant structural information from the images, e.g., diameter, area, volume and circularity. While several image analysis algorithms are available for spheroid segmentation, they all focus on compact MCTS with clearly distinguishable outer rim throughout growth. However, treated MCTS may partly be detached and destroyed and are usually obscured by dead cell debris. We successfully train two Fully Convolutional Networks, UNet and HRNet, and optimize their hyperparameters to develop an automatic segmentation for both untreated and treated MCTS. We systematically validate the automatic segmentation on larger, independent data sets of spheroids derived from two human head-and-neck cancer cell lines. We find an excellent overlap between manual and automatic segmentation for most images, quantified by Jaccard indices at around 90%. For images with smaller overlap of the segmentations, we demonstrate that this error is comparable to the variations across segmentations from different biological experts, suggesting that these images represent biologically unclear or ambiguous cases.

Figures (22)

• Figure 1: Representative examples of images for automatic segmentation with the optimized U-Net (blue) compared to the manual segmentation (green) for 3D tumor spheroids after treatment. The overlap with the manual segmentation is excellent for standard size and larger spheroids obscured by cell debris (top row) and sufficient for small, heavily obscured spheroids (bottom row). Displayed are (a,d) the original images, which are also the input for the U-Net, (b,d) magnified image details around the spheroids as indicated by the white box in (a,d) for visibility, and (c,e) corresponding contours from the segmentations. The metrics for evaluation of the automatic segmentation are: the top row - $= 7.6\%$, $= 3.7\%$ and $= 1.6\%$; bottom row - $= 22.7\%$, $= 7.5\%$ and $= 14.7\%$, see text for details.
• Figure 2: Exemplary evaluation of the automatic segmentation (blue) with respect to the manual one (green) for (a) standard case and (b),(c) rare artifacts, see text for details: (a) Correctly segmented spheroid: no contribution to or ; , and are well defined. (b) Excessive spheroids are detected beyond the actual spheroid: no contribution to , one count added to ; , and are well defined and computed for the larger, upper spheroid. (c) No overlap between automatic and manual segmentation: one count added to , no contribution to ; , and are set to one.
• Figure 3: Post-processing steps for the output of the model to transform probability heatmap to spheroid contour: (a) Probability heatmap as output of the model. Each pixel takes probability values between 0 (black) and 1 (white) predicting the target (spheroid). Note that due to the steep gradient the gradual change from black to white is hard to see. (b) By using a threshold of 0.5, the pixels are classified into outside spheroid (black) and inside spheroid (white). (c) Contour of the spheroid border extracted as a polygonal chain (blue line displayed on original image).
• Figure 4: Validation of the automatic segmentation with the optimized U-Net on larger, independent data sets shows high accuracy for the majority of cases. (a) and (b) average radial error $\Delta r$ over diameter of the manually segmented (target) spheroid $d_T$ for 6574 images of FaDu (blue triangles) and SAS (green stars) spheroids treated with different combinations and doses of X-ray irradiation and hyperthermia CheMicEckWonMenKraMcLKun2021. Manual segmentation is performed by a second biological expert (human H2, blue triangles and green stars) independently from the manual segmentation (human H1, red dots) of the training, validation and test data sets. (Results for $104$ images of test data set is displayed as red dots for comparison.) Note that the segmentation is developed only based on images of FaDu spheroids. The majority of deviations are small ($<0.2$, $\Delta r < 20\ \mu$m, red horizontal lines as guide to the eye), average (median) values are $= 0.17 (0.09) \pm 0.2$, $\Delta r = 25 (15) \pm 32\ \mu$m for the whole data set and $= 0.1 (0.07) \pm 0.09$, $\Delta r = 18 (13) \pm 17 \ \mu$m for spheroids larger $d_T\geq400\ \mu$m (red vertical lines as guide to the eye) than the initial, standard size of spheroids. Larger imprecisions for smaller spheroids are due to biologically difficult, unclear, or ambiguous cases, see text.
• Figure 5: For treated spheroids smaller than the initial, standard size before treatment ($d\leq 400\ \mu$m) deviations from the manual segmentation are not higher than variations across segmentations by different humans, suggesting that the segmentation of images with small spheroids surrounded by heavy debris is often difficult or ambiguous: Compared are segmentations from the optimized U-Net and 4 independent human experts (H2-H5) for the same 101 images, which are randomly selected from the pool of small spheroids of the extended Hold-out test data set, see fig:validation. The Friedman test score of all s 217.2 ($p \ll 0.001$) indicates significant differences among the pairwise segmentation deviations. The order according to the average JCD is (from low to high values) H5$\sim$U-Net, H3$\sim$U-Net, H3$\sim$H5, H2$\sim$H5 $<$ H4$\sim$H5, H4$\sim$U-Net, H2$\sim$U-Net, H3$\sim$H4, H2$\sim$H4, H2$\sim$H3, where the $<$ indicates a significant ($p<0.005$) difference between the sets of s according to a Dunn-Bonferroni pairwise post-hoc test.