A Language Vision Model Approach for Automated Tumor Contouring in Radiation Oncology

TL;DR

This work addresses the critical problem of accurate lung tumor contouring for radiotherapy, especially in low-resource settings where expert availability is limited. It introduces OCC, a two-stage system that pairs a Retina-UNet3D candidate detector with a Language Vision Model-based false positive reducer (GPT-4V) that fuses CT imagery and clinician text (EHR-derived descriptions) to improve contouring precision. The authors develop and test medical language vision prompts to mitigate LVM hallucinations and demonstrate improvements in false discovery rate, false positives per scan, and F1-score on a diverse CT dataset, highlighting potential for scalable, remote expert collaboration. The study suggests significant practical impact for global health equity in radiation oncology by enabling remote expert-guided, text-informed contouring that supports efficient radiotherapy planning in LMICs, while outlining limitations and directions for broader validation.

Abstract

Background: Lung cancer ranks as the leading cause of cancer-related mortality worldwide. The complexity of tumor delineation, crucial for radiation therapy, requires expertise often unavailable in resource-limited settings. Artificial Intelligence(AI), particularly with advancements in deep learning (DL) and natural language processing (NLP), offers potential solutions yet is challenged by high false positive rates. Purpose: The Oncology Contouring Copilot (OCC) system is developed to leverage oncologist expertise for precise tumor contouring using textual descriptions, aiming to increase the efficiency of oncological workflows by combining the strengths of AI with human oversight. Methods: Our OCC system initially identifies nodule candidates from CT scans. Employing Language Vision Models (LVMs) like GPT-4V, OCC then effectively reduces false positives with clinical descriptive texts, merging textual and visual data to automate tumor delineation, designed to elevate the quality of oncology care by incorporating knowledge from experienced domain experts. Results: Deployments of the OCC system resulted in a significant reduction in the false discovery rate by 35.0%, a 72.4% decrease in false positives per scan, and an F1-score of 0.652 across our dataset for unbiased evaluation. Conclusions: OCC represents a significant advance in oncology care, particularly through the use of the latest LVMs to improve contouring results by (1) streamlining oncology treatment workflows by optimizing tumor delineation, reducing manual processes; (2) offering a scalable and intuitive framework to reduce false positives in radiotherapy planning using LVMs; (3) introducing novel medical language vision prompt techniques to minimize LVMs hallucinations with ablation study, and (4) conducting a comparative analysis of LVMs, highlighting their potential in addressing medical language vision challenges.

Figures (4)

• Figure 1: OCC workflow for LMIC patients: Individuals in LMICs initially undergo diagnostic lab tests including CT scans and pathology biopsies, which are then uploaded to our OCC system. Remote domain expertsDomain experts review these scans and compile a clinical description of the findings. This narrative, together with the original CT images, is subsequently uploaded to the OCC system. Utilizing this comprehensive data, the system precisely contours true positiveTP nodules, facilitating targeted and effective radiotherapy planning.
• Figure 2: Components of the OCC model, the Candidate Tumor Detection component is fully trainable (heated), while the LVM, which serves as the False Positive Reduction model is frozen.
• Figure 3: The figure illustrates several medical vision language prompt methods. To simplify analysis, masks were combined into a single image, transitioning from a single vision input to multiple vision inputs. To conceal medical intent, the CT chest wall background was removed. A color reference was added in the top left corner for vision instructions, and contrast was enhanced while the marginal background was removed to highlight areas of interest.
• Figure 4: Cases of false positive reduction in the OCC system with different LVMs. (a) GPT-4V model accurately identifies the small TP nodule, whereas Claude3 Sonnet provides no response, and ViLT erroneously classifies it as an FP. (b) ViLT mistakenly labels the nodule as a TP, while both GPT-4V and Claude3 Sonnet correctly identify and eliminate the FP nodule.