CancerGUIDE: Cancer Guideline Understanding via Internal Disagreement Estimation

TL;DR

This paper tackles the challenge of translating complex NSCLC patient data into NCCN guideline-adherent treatment trajectories using a large language model (LLM) framework. It introduces a rigorously annotated NSCLC NSCLC dataset and a hybrid evaluation strategy that combines synthetic data, self- and cross-model consistency, and proxy benchmarks to enable zero-label performance assessment. A meta-classifier leveraging consistency signals achieves calibrated confidence (AUROC up to $0.804$) for treatment accuracy, supporting transparent risk communication and regulatory compliance. The work demonstrates that consistency-based signals can predict model correctness and that proxy benchmarks can substitute ground-truth labels at scale, offering a scalable path toward clinically viable automated decision support with controllable accuracy and interpretability.

Abstract

The National Comprehensive Cancer Network (NCCN) provides evidence-based guidelines for cancer treatment. Translating complex patient presentations into guideline-compliant treatment recommendations is time-intensive, requires specialized expertise, and is prone to error. Advances in large language model (LLM) capabilities promise to reduce the time required to generate treatment recommendations and improve accuracy. We present an LLM agent-based approach to automatically generate guideline-concordant treatment trajectories for patients with non-small cell lung cancer (NSCLC). Our contributions are threefold. First, we construct a novel longitudinal dataset of 121 cases of NSCLC patients that includes clinical encounters, diagnostic results, and medical histories, each expertly annotated with the corresponding NCCN guideline trajectories by board-certified oncologists. Second, we demonstrate that existing LLMs possess domain-specific knowledge that enables high-quality proxy benchmark generation for both model development and evaluation, achieving strong correlation (Spearman coefficient r=0.88, RMSE = 0.08) with expert-annotated benchmarks. Third, we develop a hybrid approach combining expensive human annotations with model consistency information to create both the agent framework that predicts the relevant guidelines for a patient, as well as a meta-classifier that verifies prediction accuracy with calibrated confidence scores for treatment recommendations (AUROC=0.800), a critical capability for communicating the accuracy of outputs, custom-tailoring tradeoffs in performance, and supporting regulatory compliance. This work establishes a framework for clinically viable LLM-based guideline adherence systems that balance accuracy, interpretability, and regulatory requirements while reducing annotation costs, providing a scalable pathway toward automated clinical decision support.

Figures (7)

• Figure 1: CancerGUIDE framework. (1) Clinicians derive patient pathways ($label_{doc}$) from NCCN guidelines and real notes. These serve as gold-standard references to compare with LLM-derived pathways ($label_{LLM}$), producing a reference accuracy score ($Score_r$). (2) NCCN guidelines and both synthetic and real clinical notes are used to generate weak labels ($label_{proxy}$). LLM predictions ($label_{LLM}$) are compared to these proxy labels to compute proxy performance scores ($Score_p$), enabling evaluation of how well synthetic supervision approximates expert annotations. (3) Model consistency features (Feature Sets) and $Score_p$ are used to train a logistic regression meta-classifier that predicts whether a treatment recommendation is likely correct. This classifier is fit on labelled data from step 1 and unlabelled features from step 2. The classifier is applied at test time to accept or reject LLM-generated recommendations, supporting confidence estimation and threshold selection for clinical deployment.
• Figure 2: Self-consistency pseudo-labels provide a robust proxy for benchmarking. Six approaches are evaluated: synthetic data (structured and unstructured), self-consistency pseudo-labeling (with varying acceptance criteria), and cross-model consistency pseudo-labeling. Correlation is measured using Spearman coefficients as well as root mean-squared error with color intensity indicating magnitude.
• Figure 3: Model accuracy increases with self-consistency across prediction runs. Higher consistency (fraction of runs producing identical paths) correlates with improved performance for treatment matching.
• Figure 4: Using signals from self- and cross-model consistency provides high AUC for accuracy classification. Proxy benchmark results do not provide significant prediction signal as compared to consistency. Model AUCs range from 0.703 to 0.981, indicating strong prediction capability. LLaMA-3.3-70B-Instruct's high AUC can be attributed to its low performance on the given task, creating an arbitrary classification problem and highlighting limitations of including lower-performing models in analyses.
• Figure 5: Identification of most common discrepancies between human annotations and model annotations compared to most common discrepancies between $k$ model rollouts of path prediction.