Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Adverse Drug Event Detection with Multimodal Dataset: Corpus Creation and Model Development

Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Aman Chadha, Samrat Mondal

TL;DR

This work presents a MultiModal Adverse Drug Event (MMADE) detection dataset, and introduces a framework that leverages the capabilities of LLMs and VLMs for ADE detection by generating detailed descriptions of medical images depicting ADEs, aiding healthcare professionals in visually identifying adverse events.

Abstract

The mining of adverse drug events (ADEs) is pivotal in pharmacovigilance, enhancing patient safety by identifying potential risks associated with medications, facilitating early detection of adverse events, and guiding regulatory decision-making. Traditional ADE detection methods are reliable but slow, not easily adaptable to large-scale operations, and offer limited information. With the exponential increase in data sources like social media content, biomedical literature, and Electronic Medical Records (EMR), extracting relevant ADE-related information from these unstructured texts is imperative. Previous ADE mining studies have focused on text-based methodologies, overlooking visual cues, limiting contextual comprehension, and hindering accurate interpretation. To address this gap, we present a MultiModal Adverse Drug Event (MMADE) detection dataset, merging ADE-related textual information with visual aids. Additionally, we introduce a framework that leverages the capabilities of LLMs and VLMs for ADE detection by generating detailed descriptions of medical images depicting ADEs, aiding healthcare professionals in visually identifying adverse events. Using our MMADE dataset, we showcase the significance of integrating visual cues from images to enhance overall performance. This approach holds promise for patient safety, ADE awareness, and healthcare accessibility, paving the way for further exploration in personalized healthcare.

Enhancing Adverse Drug Event Detection with Multimodal Dataset: Corpus Creation and Model Development

TL;DR

This work presents a MultiModal Adverse Drug Event (MMADE) detection dataset, and introduces a framework that leverages the capabilities of LLMs and VLMs for ADE detection by generating detailed descriptions of medical images depicting ADEs, aiding healthcare professionals in visually identifying adverse events.

Abstract

The mining of adverse drug events (ADEs) is pivotal in pharmacovigilance, enhancing patient safety by identifying potential risks associated with medications, facilitating early detection of adverse events, and guiding regulatory decision-making. Traditional ADE detection methods are reliable but slow, not easily adaptable to large-scale operations, and offer limited information. With the exponential increase in data sources like social media content, biomedical literature, and Electronic Medical Records (EMR), extracting relevant ADE-related information from these unstructured texts is imperative. Previous ADE mining studies have focused on text-based methodologies, overlooking visual cues, limiting contextual comprehension, and hindering accurate interpretation. To address this gap, we present a MultiModal Adverse Drug Event (MMADE) detection dataset, merging ADE-related textual information with visual aids. Additionally, we introduce a framework that leverages the capabilities of LLMs and VLMs for ADE detection by generating detailed descriptions of medical images depicting ADEs, aiding healthcare professionals in visually identifying adverse events. Using our MMADE dataset, we showcase the significance of integrating visual cues from images to enhance overall performance. This approach holds promise for patient safety, ADE awareness, and healthcare accessibility, paving the way for further exploration in personalized healthcare.
Paper Structure (25 sections, 9 figures, 5 tables)

This paper contains 25 sections, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Corpus Development
  4. Data Collection
  5. Social Media
  6. Healthcare Blog
  7. Medical Case Reports
  8. Data Annotation
  9. Annotation Manual
  10. Corpus Analysis
  11. Problem Formulation
  12. Methodology
  13. Experimental Results and Analysis
  14. Findings from Experiments
  15. Qualitative Analysis
  16. ...and 10 more sections

Figures (9)

  • Figure 1: Samples from the dataset highlight the significance of visual cues in understanding adverse drug events, particularly in cases where patients are unaware of a specific medical condition.
  • Figure 2: Some samples from the dataset containing images and the corresponding descriptive text.
  • Figure 3: Distribution of different body parts in the curated MMADE dataset. The number of data points and the percentage corresponding to each category have been provided.
  • Figure 4: Data distribution statistics from various sources, illustrating the total ADE data (images, text, and image-text pairs), and the relevant image-text pairs.
  • Figure 5: Illustration of an annotation process, with A1, A2, and A3 representing individual annotators and MV signifying final majority voting. In this setup, "1" denotes ADR, while "0" is unrelated to ADR.
  • ...and 4 more figures