Large-Language-Model Empowered Dose Volume Histogram Prediction for Intensity Modulated Radiotherapy

TL;DR

The paper addresses DVH prediction for IMRT planning by converting CT images into a structured image-dose graph and applying DoseGNN, a graph neural network that fuses image features with clinician prescriptions via LLM-driven prompt nodes. An online Human-AI Collaboration (OHAC) framework enables clinician guidance and continual model refinement, aiming to overcome data scarcity and enhance plan quality. DoseGNN, especially when augmented with LLM prompts, significantly reduces prediction error compared with strong deep-learning baselines and better matches delivered dose distributions, as evidenced by MSE and CDVH analyses. This approach offers a practical path toward faster, clinician-in-the-loop automatic planning and potential multi-center collaboration for knowledge-based planning in radiotherapy.

Abstract

Treatment planning is currently a patient specific, time-consuming, and resource demanding task in radiotherapy. Dose-volume histogram (DVH) prediction plays a critical role in automating this process. The geometric relationship between DVHs in radiotherapy plans and organs-at-risk (OAR) and planning target volume (PTV) has been well established. This study explores the potential of deep learning models for predicting DVHs using images and subsequent human intervention facilitated by a large-language model (LLM) to enhance the planning quality. We propose a pipeline to convert unstructured images to a structured graph consisting of image-patch nodes and dose nodes. A novel Dose Graph Neural Network (DoseGNN) model is developed for predicting DVHs from the structured graph. The proposed DoseGNN is enhanced with the LLM to encode massive knowledge from prescriptions and interactive instructions from clinicians. In this study, we introduced an online human-AI collaboration (OHAC) system as a practical implementation of the concept proposed for the automation of intensity-modulated radiotherapy (IMRT) planning. In comparison to the widely-employed DL models used in radiotherapy, DoseGNN achieved mean square errors that were 80$\%$, 76$\%$ and 41.0$\%$ of those predicted by Swin U-Net Transformer, 3D U-Net CNN and vanilla MLP, respectively. Moreover, the LLM-empowered DoseGNN model facilitates seamless adjustment to treatment plans through interaction with clinicians using natural language.

Figures (5)

• Figure 1: . Illustration of image conversion pipeline and DoseGNN model. The image conversion pipeline converts unstructured 3D images to structured 4D tensor/data that summarizes the contour information and geometric relation of OARs and PTVs. In DoseGNN, Image processing deep learning models (like swin Transform) and LLMs are used to convert the structured 4D tensor and prescriptions/instructions text to node features in an image-dose graph, then a GNN is applied to predict dose values of dose nodes in the graph.
• Figure 2: Comparison of predictive performance of deep learning models in DVH prediction.
• Figure 3: Comparison of predicted cumulative DVH (CDVH) against the CDVH of delivered dose
• Figure 4: Clinicians can directly optimize the outcome of DoseGNN model with prompts/ instructions. In the figure, dashed blue lines indicate CDVHs computed from unknown delivered doses in clinical plans designed/optimized by human experts. A low-quality prediction of doses shows significantly different shape of CDVH and can be identified by doctors and other expert users without knowing the dashed blue lines.
• Figure 5: Illustration of the OHAC (online Human-AI collaboration) system.

Theorems & Definitions (1)

• Definition 2.1