Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adapting Biomedical Abstracts into Plain language using Large Language Models

Haritha Gangavarapu, Giridhar Kaushik Ramachandran, Kevin Lybarger, Meliha Yetisgen, Özlem Uzuner

TL;DR

The paper presents PLABA, a benchmark and methodology for converting biomedical abstracts into plain language tailored to the general public. Using a dataset of 750 PubMed abstracts and 921 sentence-level adaptations derived from 75 MedlinePlus questions, the authors compare PEFT-fine-tuned LLaMa2 models with zero-shot and few-shot prompting of GPT-3.5/GPT-4, as well as T5-based baselines. Human evaluation reveals that a GPT-4–based one-shot in-context approach achieves the best simplicity and strong faithfulness, while LLaMa2 yields strong automatic metric performance. Overall, the work demonstrates that large language models, particularly GPT-4 in ICL and instruction-guided settings, can effectively make biomedical information more accessible, with PLABA serving as a valuable benchmark for future development.

Abstract

A vast amount of medical knowledge is available for public use through online health forums, and question-answering platforms on social media. The majority of the population in the United States doesn't have the right amount of health literacy to make the best use of that information. Health literacy means the ability to obtain and comprehend the basic health information to make appropriate health decisions. To build the bridge between this gap, organizations advocate adapting this medical knowledge into plain language. Building robust systems to automate the adaptations helps both medical and non-medical professionals best leverage the available information online. The goal of the Plain Language Adaptation of Biomedical Abstracts (PLABA) track is to adapt the biomedical abstracts in English language extracted from PubMed based on the questions asked in MedlinePlus for the general public using plain language at the sentence level. As part of this track, we leveraged the best open-source Large Language Models suitable and fine-tuned for dialog use cases. We compare and present the results for all of our systems and our ranking among the other participants' submissions. Our top performing GPT-4 based model ranked first in the avg. simplicity measure and 3rd on the avg. accuracy measure.

Adapting Biomedical Abstracts into Plain language using Large Language Models

TL;DR

The paper presents PLABA, a benchmark and methodology for converting biomedical abstracts into plain language tailored to the general public. Using a dataset of 750 PubMed abstracts and 921 sentence-level adaptations derived from 75 MedlinePlus questions, the authors compare PEFT-fine-tuned LLaMa2 models with zero-shot and few-shot prompting of GPT-3.5/GPT-4, as well as T5-based baselines. Human evaluation reveals that a GPT-4–based one-shot in-context approach achieves the best simplicity and strong faithfulness, while LLaMa2 yields strong automatic metric performance. Overall, the work demonstrates that large language models, particularly GPT-4 in ICL and instruction-guided settings, can effectively make biomedical information more accessible, with PLABA serving as a valuable benchmark for future development.

Abstract

A vast amount of medical knowledge is available for public use through online health forums, and question-answering platforms on social media. The majority of the population in the United States doesn't have the right amount of health literacy to make the best use of that information. Health literacy means the ability to obtain and comprehend the basic health information to make appropriate health decisions. To build the bridge between this gap, organizations advocate adapting this medical knowledge into plain language. Building robust systems to automate the adaptations helps both medical and non-medical professionals best leverage the available information online. The goal of the Plain Language Adaptation of Biomedical Abstracts (PLABA) track is to adapt the biomedical abstracts in English language extracted from PubMed based on the questions asked in MedlinePlus for the general public using plain language at the sentence level. As part of this track, we leveraged the best open-source Large Language Models suitable and fine-tuned for dialog use cases. We compare and present the results for all of our systems and our ranking among the other participants' submissions. Our top performing GPT-4 based model ranked first in the avg. simplicity measure and 3rd on the avg. accuracy measure.
Paper Structure (14 sections, 2 figures, 4 tables)

This paper contains 14 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Dataset
  5. Plain language adaptation using LLMs
  6. Experimental setup
  7. Evaluation
  8. Evaluation on internal test set
  9. External evaluation on the task test set
  10. Results and Discussion
  11. Automatic evaluation
  12. Human evaluation
  13. Conclusions
  14. Acknowledgements

Figures (2)

  • Figure 1: elements of the Prompt structure for Fine-tuning LLaMa-2 models
  • Figure 2: Prompt structure for the ICL using GPT-4 and Fine-tuning GPT-3.5