Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AlpaPICO: Extraction of PICO Frames from Clinical Trial Documents Using LLMs

Madhusudan Ghosh, Shrimon Mukherjee, Asmit Ganguly, Partha Basuchowdhuri, Sudip Kumar Naskar, Debasis Ganguly

TL;DR

Both of the proposed frameworks utilize AlpaCare as base LLM which employs both few-shot in-context learning and instruction tuning techniques to extract PICO-related terms from the clinical trial reports, and applied these approaches to the widely used coarse-grained datasets such as EBM-NLP, EBM-COMET and fine-grained datasets such as EBM-NLPrev and EBM-NLPh.

Abstract

In recent years, there has been a surge in the publication of clinical trial reports, making it challenging to conduct systematic reviews. Automatically extracting Population, Intervention, Comparator, and Outcome (PICO) from clinical trial studies can alleviate the traditionally time-consuming process of manually scrutinizing systematic reviews. Existing approaches of PICO frame extraction involves supervised approach that relies on the existence of manually annotated data points in the form of BIO label tagging. Recent approaches, such as In-Context Learning (ICL), which has been shown to be effective for a number of downstream NLP tasks, require the use of labeled examples. In this work, we adopt ICL strategy by employing the pretrained knowledge of Large Language Models (LLMs), gathered during the pretraining phase of an LLM, to automatically extract the PICO-related terminologies from clinical trial documents in unsupervised set up to bypass the availability of large number of annotated data instances. Additionally, to showcase the highest effectiveness of LLM in oracle scenario where large number of annotated samples are available, we adopt the instruction tuning strategy by employing Low Rank Adaptation (LORA) to conduct the training of gigantic model in low resource environment for the PICO frame extraction task. Our empirical results show that our proposed ICL-based framework produces comparable results on all the version of EBM-NLP datasets and the proposed instruction tuned version of our framework produces state-of-the-art results on all the different EBM-NLP datasets. Our project is available at \url{https://github.com/shrimonmuke0202/AlpaPICO.git}.

AlpaPICO: Extraction of PICO Frames from Clinical Trial Documents Using LLMs

TL;DR

Both of the proposed frameworks utilize AlpaCare as base LLM which employs both few-shot in-context learning and instruction tuning techniques to extract PICO-related terms from the clinical trial reports, and applied these approaches to the widely used coarse-grained datasets such as EBM-NLP, EBM-COMET and fine-grained datasets such as EBM-NLPrev and EBM-NLPh.

Abstract

In recent years, there has been a surge in the publication of clinical trial reports, making it challenging to conduct systematic reviews. Automatically extracting Population, Intervention, Comparator, and Outcome (PICO) from clinical trial studies can alleviate the traditionally time-consuming process of manually scrutinizing systematic reviews. Existing approaches of PICO frame extraction involves supervised approach that relies on the existence of manually annotated data points in the form of BIO label tagging. Recent approaches, such as In-Context Learning (ICL), which has been shown to be effective for a number of downstream NLP tasks, require the use of labeled examples. In this work, we adopt ICL strategy by employing the pretrained knowledge of Large Language Models (LLMs), gathered during the pretraining phase of an LLM, to automatically extract the PICO-related terminologies from clinical trial documents in unsupervised set up to bypass the availability of large number of annotated data instances. Additionally, to showcase the highest effectiveness of LLM in oracle scenario where large number of annotated samples are available, we adopt the instruction tuning strategy by employing Low Rank Adaptation (LORA) to conduct the training of gigantic model in low resource environment for the PICO frame extraction task. Our empirical results show that our proposed ICL-based framework produces comparable results on all the version of EBM-NLP datasets and the proposed instruction tuned version of our framework produces state-of-the-art results on all the different EBM-NLP datasets. Our project is available at \url{https://github.com/shrimonmuke0202/AlpaPICO.git}.
Paper Structure (29 sections, 2 equations, 9 figures, 8 tables)

This paper contains 29 sections, 2 equations, 9 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Our Contributions
  3. Related Work
  4. General NER Techniques
  5. Evidence-based Medical Entity Extraction
  6. Large Language Models and In-context Learning
  7. Instruction tuning
  8. Methodology
  9. Overview of Sequence Labeling task
  10. Overview of In-Context Learning
  11. Task Description
  12. Demonstrations
  13. Extractions
  14. Architecture
  15. Retrieval-based approach to obtain relevant context for Entity Recognition Task
  16. ...and 14 more sections

Figures (9)

  • Figure 1: Example of PICO frame extraction using our proposed framework AlpaPICO. Here we pass documents to our proposed framework for extracting PICO frames from the clinical trial documents.
  • Figure 2: Visuaization of our proposed ICL-based framework for PICO frame extraction
  • Figure 3: Formats of input and output of our in-context learning based framework for name entity recognition task. The input is formed by the Task Description, Demonstrations, and the Input Sentence.
  • Figure 4: LoRA block of our AlpaPICO framework.
  • Figure 5: The data construction prompt is utilized to produce entity mentions and their corresponding types for a specific passage.
  • ...and 4 more figures