Large Language Models for Cancer Communication: Evaluating Linguistic Quality, Safety, and Accessibility in Generative AI

TL;DR

The study evaluates eight LLMs—five general-purpose and three medical-domain—on breast and cervical cancer communication using a three-dimensional framework of linguistic quality, safety/trustworthiness, and accessibility/affectiveness. It blends quantitative metrics (BLEURT, BERTScore, ROUGE, toxicity, bias, readability) with qualitative expert ratings across 400 prompts to compare models under realistic patient-communication tasks. Key findings show general-purpose LLMs yield higher linguistic quality and affective resonance, while medical LLMs offer better accessibility yet exhibit higher harm and bias potential, indicating a trade-off between domain knowledge and safety. The work argues for hybrid, knowledge-infused architectures and grounding techniques to produce accurate, safe, and accessible cancer information at scale, informing future AI-enabled digital health tools.

Abstract

Effective communication about breast and cervical cancers remains a persistent health challenge, with significant gaps in public understanding of cancer prevention, screening, and treatment, potentially leading to delayed diagnoses and inadequate treatments. This study evaluates the capabilities and limitations of Large Language Models (LLMs) in generating accurate, safe, and accessible cancer-related information to support patient understanding. We evaluated five general-purpose and three medical LLMs using a mixed-methods evaluation framework across linguistic quality, safety and trustworthiness, and communication accessibility and affectiveness. Our approach utilized quantitative metrics, qualitative expert ratings, and statistical analysis using Welch's ANOVA, Games-Howell, and Hedges' g. Our results show that general-purpose LLMs produced outputs of higher linguistic quality and affectiveness, while medical LLMs demonstrate greater communication accessibility. However, medical LLMs tend to exhibit higher levels of potential harm, toxicity, and bias, reducing their performance in safety and trustworthiness. Our findings indicate a duality between domain-specific knowledge and safety in health communications. The results highlight the need for intentional model design with targeted improvements, particularly in mitigating harm and bias, and improving safety and affectiveness. This study provides a comprehensive evaluation of LLMs for cancer communication, offering critical insights for improving AI-generated health content and informing future development of accurate, safe, and accessible digital health tools.

Figures (3)

• Figure 1: Overview of Datasets and Large Language Models Used for Evaluating Breast and Cervical Cancer QA Tasks
• Figure 2: Comprehensive evaluation framework for evaluating general purpose and specialized medical LLMs for cancer communication.
• Figure 3: Similarity scores between responses without context and responses with context (e.g., African American, Female, Hispanic). This shows that general-purpose models like Llama 3 and Gemma consistently maintain high similarity scores across demographic contexts, indicating lower bias and stronger demographic consistency. In contrast, medical LLMs such as BioMistral and MedAlpaca display greater variability and lower similarity scores, especially across race, ethnicity, and language background. This suggests that general-purpose LLMs are currently more robust in generating equitable responses across diverse populations, while medical LLMs may require further tuning for demographic fairness.