Building Trust in Clinical LLMs: Bias Analysis and Dataset Transparency

TL;DR

This work addresses the risk that pretraining data biases propagate into clinical LLMs by introducing HC4, a healthcare-focused corpus exceeding 89 billion tokens, and a bias-evaluation framework tailored to healthcare. It combines general-domain bias benchmarks (e.g., BOLD) with a healthcare-specific probe for differential opioid prescribing, implemented through nine models across GPT-2, Llama-3, and Mistral trained on HC4, SlimPajama, and FineWeb. The study demonstrates that pretraining data and architecture shape bias patterns, revealing distinct healthcare biases such as ethnicity- and age-related prescribing tendencies, and highlights the Net Bias Prescription Score $NBPS = M_{over} - M_{under}$ as a key metric. By releasing HC4 and a transparent bias-analysis methodology, the paper advocates for systematic, domain-aware bias evaluation throughout dataset curation and model deployment to promote fair and safe clinical AI.

Abstract

Large language models offer transformative potential for healthcare, yet their responsible and equitable development depends critically on a deeper understanding of how training data characteristics influence model behavior, including the potential for bias. Current practices in dataset curation and bias assessment often lack the necessary transparency, creating an urgent need for comprehensive evaluation frameworks to foster trust and guide improvements. In this study, we present an in-depth analysis of potential downstream biases in clinical language models, with a focus on differential opioid prescription tendencies across diverse demographic groups, such as ethnicity, gender, and age. As part of this investigation, we introduce HC4: Healthcare Comprehensive Commons Corpus, a novel and extensively curated pretraining dataset exceeding 89 billion tokens. Our evaluation leverages both established general benchmarks and a novel, healthcare-specific methodology, offering crucial insights to support fairness and safety in clinical AI applications.

Figures (11)

• Figure 1: Sentiment distribution shifts from Wikipedia baseline across different language models and pretraining datasets for ethnicity groups. Each subplot represents a specific model (GPT-2, LLaMA-3.2, Mistral) and dataset (HC4, SP, FW) combination. Bars show the difference in sentiment proportions (negative, neutral, positive) between model-generated completions and Wikipedia baseline texts for different ethnicity groups. Positive values indicate higher proportion in generated text compared to baseline, while negative values indicate lower proportion. Asterisks (*) denote statistically significant shifts after FDR correction ($p_{corr} < 0.05$). The analysis reveals systematic differences in how language models portray different genders compared to the original Wikipedia distribution, with notable variations across models and datasets. HC4: Healthcare Comprehensive Commons Corpus; SP: SlimPajama; FW: FineWeb.
• Figure 2: Bar charts displaying ethnicity prescription bias across different model architectures (GPT-2, Llama-3, Mistral) trained on three datasets (HC4, SlimPajama, FineWeb) for opioid (left) and non-opioid medications (right). Each bar represents the Net Bias Prescription Score (NBPS), calculated as the difference between the number of medications with statistically significant higher prescription probabilities and those with statistically significant lower probabilities relative to ethnicity-neutral prompts. Positive values indicate overprescription bias, while negative values show underprescription bias. Statistical significance was determined using Wilcoxon signed-rank tests with Bonferroni correction for multiple comparisons.
• Figure 3: Validation set perplexity for three Llama-3.2 model variants, trained on three datasets.
• Figure 4: Qualitative examples of the text generated by three Llama-3.2 model variants, trained on three datasets. Black text represents the prompt and the blue text was generated by the models given that prompt. FW: FineWeb, SP: SlimPijama, HC4: our dataset.
• Figure 5: Example from the baseline BOLD Wikipedia set (above), corresponding prompt and generated text (below). The baseline sample sentiment is positive, whereas sentiment of the completion generated by the model can be different.