Vision-Language Model-Based Semantic-Guided Imaging Biomarker for Lung Nodule Malignancy Prediction

TL;DR

The study tackles the challenge of explainable, generalizable lung nodule malignancy prediction across diverse CT modalities by leveraging a vision-language framework that aligns CT nodule imaging with radiologist-derived semantic features using a pretrained CLIP model. It introduces a 2.5D CT representation and Gemini-generated radiology-text for semantic features, with LoRA-based parameter-efficient fine-tuning and dual prediction branches for imaging and semantics, plus zero-shot inference for semantic explainability. The approach achieves state-of-the-art performance on the NLST test set (AUROC 0.901, AUPRC 0.776) and maintains robust results across external datasets (LUNGx, UCLA, DLCS), demonstrating strong cross-dataset generalization and interpretability. By reducing reliance on real-time radiologist annotations at inference and providing semantically grounded explanations, the method offers practical clinical utility and scalable deployment, with code publicly available.

Abstract

Machine learning models have utilized semantic features, deep features, or both to assess lung nodule malignancy. However, their reliance on manual annotation during inference, limited interpretability, and sensitivity to imaging variations hinder their application in real-world clinical settings. Thus, this research aims to integrate semantic features derived from radiologists' assessments of nodules, guiding the model to learn clinically relevant, robust, and explainable imaging features for predicting lung cancer. We obtained 938 low-dose CT scans from the National Lung Screening Trial (NLST) with 1,261 nodules and semantic features. Additionally, the Lung Image Database Consortium dataset contains 1,018 CT scans, with 2,625 lesions annotated for nodule characteristics. Three external datasets were obtained from UCLA Health, the LUNGx Challenge, and the Duke Lung Cancer Screening. We fine-tuned a pretrained Contrastive Language-Image Pretraining (CLIP) model with a parameter-efficient fine-tuning approach to align imaging and semantic text features and predict the one-year lung cancer diagnosis. Our model outperformed state-of-the-art (SOTA) models in the NLST test set with an AUROC of 0.901 and AUPRC of 0.776. It also showed robust results in external datasets. Using CLIP, we also obtained predictions on semantic features through zero-shot inference, such as nodule margin (AUROC: 0.807), nodule consistency (0.812), and pleural attachment (0.840). Our approach surpasses the SOTA models in predicting lung cancer across datasets collected from diverse clinical settings, providing explainable outputs, aiding clinicians in comprehending the underlying meaning of model predictions. This approach also prevents the model from learning shortcuts and generalizes across clinical settings. The code is available at https://github.com/luotingzhuang/CLIP_nodule.

Figures (4)

• Figure 1: CT and Semantic Features Preprocessing Steps.
• Figure 2: CLIP Model Architecture and Fine-tuning Methods. (a) The pretrained CLIP was fine-tuned to pull the paired imaging and semantic features close together, which allowed the model to learn meaningful relationships between imaging features and semantic features. During training, nodule images sliced in nine different directions from 3D nodule crops were passed into the vision transformer image encoder. The attention-based multiple-instance learning module aggregated the output imaging features to obtain a single embedding. The sentence containing the semantic features was passed into the text encoder to generate the text feature embedding. The visual and text features were then aligned using CLIP loss. Two prediction heads were independently attached after the encoders to predict the one-year lung cancer risk. During inference, only the imaging feature was required, allowing it to be applied universally without the need for a radiologist’s evaluation of the CT scan. (b) We examined the training of the CLIP model using three distinct tuning methods as part of our ablation study. All parameters were fine-tuned (left). Only two projection layers were fine-tuned (middle). A parameter-efficient fine-tuning method, Low-Rank Adaptation (LoRA), which involves inserting trainable low-rank matrices into each layer of the vision and text encoders, was used in our final model (right). While keeping the pretrained weights frozen, we updated the low-rank matrices. We also allowed the projection layers to be fine-tuned.
• Figure 3: Error Analysis. One lung cancer and one non-lung cancer case from each dataset are presented. We present the CT scan slice corresponding to the middle of the nodule, highlighted with a red bounding box. In the top right corner, we place the magnified view of the nodule for clearer visualization. Certain features may not be fully appreciable in the single slice but are visible when viewing the whole series. Patient-level risk score from each model is shown beside each image. The scale and range of the predicted probability across different models can vary significantly. The probabilities from Sybil, Venkadesh et al., and our model have been calibrated. Therefore, the predicted probability from these three models matches the observed frequency. Sybil risk scores for NLST are listed only if the case is included in Sybil's test data.
• Figure A1: Error Analysis (Extended). One lung cancer and one non-lung cancer case from each dataset are presented. We present the CT scan slice corresponding to the middle of the nodule, highlighted with a red bounding box and magnified in the top right corner. Certain features may not be fully appreciable in the single slice but are visible when viewing the whole series. Patient-level risk scores are shown beside images.