Evaluating and Mitigating Bias in AI-Based Medical Text Generation

TL;DR

The paper tackles fairness in AI-based medical text generation by introducing a metric-aware fairness difference ($MFD$) and a selective optimization framework that prioritizes underperforming subgroups using a combination of cross-entropy loss and pathology-aware ranking losses. Evaluated across radiology report generation, report summarization, and scholarly paper summarization on the MIMIC-CXR and PubMed datasets, the approach scales from hundreds of millions to billions of parameters, including R2Gen, BART, and Llama2-13B with LoRA. Results show substantial reductions in subgroup and intersectional disparities (often >$30\%$–$35\%$ on average) while maintaining or improving overall generation quality, demonstrating practical fairness improvements without sacrificing performance. The work provides publicly available code and supports broad applicability to medical text generation and LLM-based workflows, signaling actionable steps toward equitable AI in healthcare.

Abstract

Artificial intelligence (AI) systems, particularly those based on deep learning models, have increasingly achieved expert-level performance in medical applications. However, there is growing concern that such AI systems may reflect and amplify human bias, and reduce the quality of their performance in historically under-served populations. The fairness issue has attracted considerable research interest in the medical imaging classification field, yet it remains understudied in the text generation domain. In this study, we investigate the fairness problem in text generation within the medical field and observe significant performance discrepancies across different races, sexes, and age groups, including intersectional groups, various model scales, and different evaluation metrics. To mitigate this fairness issue, we propose an algorithm that selectively optimizes those underperformed groups to reduce bias. The selection rules take into account not only word-level accuracy but also the pathology accuracy to the target reference, while ensuring that the entire process remains fully differentiable for effective model training. Our evaluations across multiple backbones, datasets, and modalities demonstrate that our proposed algorithm enhances fairness in text generation without compromising overall performance. Specifically, the disparities among various groups across different metrics were diminished by more than 30% with our algorithm, while the relative change in text generation accuracy was typically within 2%. By reducing the bias generated by deep learning models, our proposed approach can potentially alleviate concerns about the fairness and reliability of text generation diagnosis in medical domain. Our code is publicly available to facilitate further research at https://github.com/iriscxy/GenFair.

Figures (8)

• Figure 1: An example where both male and female have similar input, yet the text generation model neglects cardiomegaly in the male case.
• Figure 2: Model Pipeline: Our approach involves three tasks: radiology report generation, report summarization, and paper summarization, using the public benchmark MIMIC-CXR and PubMed datasets. We trained a deep-learning text generation model using our selective optimization mechanism, which considers two key aspects: word-level overlap and pathology observation. We evaluated pairwise fairness across different subgroups, including sex, race, and age, to determine if the model is equally fair for all individuals in each subgroup.
• Figure 3: Performance disparities were observed across various categories, including sex (female vs. male), race (black vs. white), age (young vs. aged), and species (humans vs. animals), for ROUGE comparison in: (a) radiology report generation, (b) summarization on the CXR dataset, and (c) summarization on the PubMed dataset; and for CheXpert comparison in: (d) radiology report generation and (e) report summarization. Significant differences, denoted by * (p $<$ 0.05) and ** (p $<$ 0.01), were identified using the two-side Mann-Whitney U test. Error bars represent 95% confidence intervals.
• Figure 4: Performance disparities were observed across intersectional groups in: (a) radiology report generation, and (b)(c) report summarization. The top row represents baseline performance, while the bottom row illustrates the enhanced performance achieved by incorporating our proposed method. Significant differences, denoted by * (p $<$ 0.05), ** (p $<$ 0.01), and *** (p $<$ 0.001), were identified using the two-side Mann-Whitney U test. Error bars represent 95% confidence intervals.
• Figure 5: The details of the selection algorithm. The encoder-decoder or decoder-only model outputs the generation probability for the reference as well as a set of candidates. The ranking loss evaluates if the model assigns a greater generation probability to candidates of higher quality compared to those of lower quality. Cases with a high sum of cross-entropy loss and ranking loss are selected for optimization.