PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology

TL;DR

A human-in-the-loop pipeline is developed to create expert rubrics for de-identified patient questions from the Pancreatic Cancer Action Network, and 22 proprietary and open-source LLMs using an LLM-as-a-judge framework are evaluated, measuring clinical completeness, factual accuracy, and web-search integration.

Abstract

Large language models (LLMs) have achieved expert-level performance on standardized examinations, yet multiple-choice accuracy poorly reflects real-world clinical utility and safety. As patients and clinicians increasingly use LLMs for guidance on complex conditions such as pancreatic cancer, evaluation must extend beyond general medical knowledge. Existing frameworks, such as HealthBench, rely on simulated queries and lack disease-specific depth. Moreover, high rubric-based scores do not ensure factual correctness, underscoring the need to assess hallucinations. We developed a human-in-the-loop pipeline to create expert rubrics for de-identified patient questions from the Pancreatic Cancer Action Network (PanCAN). The resulting benchmark, PanCanBench, includes 3,130 question-specific criteria across 282 authentic patient questions. We evaluated 22 proprietary and open-source LLMs using an LLM-as-a-judge framework, measuring clinical completeness, factual accuracy, and web-search integration. Models showed substantial variation in rubric-based completeness, with scores ranging from 46.5% to 82.3%. Factual errors were common, with hallucination rates (the percentages of responses containing at least one factual error) ranging from 6.0% for Gemini-2.5 Pro and GPT-4o to 53.8% for Llama-3.1-8B. Importantly, newer reasoning-optimized models did not consistently improve factuality: although o3 achieved the highest rubric score, it produced inaccuracies more frequently than other GPT-family models. Web-search integration did not inherently guarantee better responses. The average score changed from 66.8% to 63.9% for Gemini-2.5 Pro and from 73.8% to 72.8% for GPT-5 when web search was enabled. Synthetic AI-generated rubrics inflated absolute scores by 17.9 points on average while generally maintaining similar relative ranking.

Figures (10)

• Figure 1: Overview of how we use rubrics to evaluate large language model (LLM) responses to patient and caregiver questions.a) Collect de-identified patient and caregiver questions from the Pancreatic Cancer Action Network. b) Generate responses from LLMs. c) Grade the LLM responses using an AI judge operating under human-curated evaluation rubrics. d) Factual error analysis using LLM-as-a-judge.
• Figure 2: Pipeline to create rubrics and evaluate responses to Pancreatic Cancer Action Network (PanCAN) hotline questions guided by human expertise.Phase 1) Two oncology fellows independently designed initial rubrics. Phase 2) Leverage AI to polish the human-written rubrics and perform ‘gut checks’ on the rubrics by comparing the scores of pairs of large and small models in the same family. Phase 3) AI merged the rubrics from the oncology fellows by semantically joining redundant rubric items and retaining unique items. Phase 4) A third oncology fellow reviewed and finalized the rubrics.
• Figure 3: Pipeline to create rubrics and evaluate responses to Pancreatic Cancer Action Network (PanCAN) hotline questions guided by human expertise.a) Frequency of questions in each category b) Example questions for each question category.
• Figure 4: Comprehensive benchmark study across 22 models.a) Barplot showing average score by model. b) UpSet plot showing the percentage of responses containing factual errors and the question categories where each model is most prone to mistakes. c) Average scores for questions with rubric items requiring references/citations, with and without web search. d) Table showing triggering rate, success rate when web search is used, key failure mode, and supportive link rate for GPT-5, Claude-Sonnet-4.5, and Gemini-2.5 pro with web search function.
• Figure 5: Investigating the necessity of human involvement in LLM Evaluation.a) Paired barplot showing average score by model evaluated by human-curated versus AI-generated synthetic rubrics. b) Slope plot showing the ranking comparison of models evaluated by human rubrics and AI-generated rubrics.