Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Free Form Medical Visual Question Answering in Radiology

Abhishek Narayanan, Rushabh Musthyala, Rahul Sankar, Anirudh Prasad Nistala, Pranav Singh, Jacopo Cirrone

TL;DR

The paper tackles free-form medical VQA in radiology by addressing data scarcity with targeted dataset augmentation and by leveraging intra-domain transfer learning through radiology-pretrained image encoders. It introduces MedCLiP-based cross-modal supervision to learn robust multi-modal representations and frames VQA as a classification task to streamline benchmarking on SLAKE. Empirical results show intra-domain pretraining with radiology data yields strong performance, with the MedCLiP approach achieving a top-1 accuracy of 79.55% on SLAKE, comparable to state-of-the-art while using a simpler architecture. The work highlights practical implications for diagnostic-support tools, though it notes limitations in explainability and vocabulary scope that warrant future improvements in interpretability and open-ended responses.

Abstract

Visual Question Answering (VQA) in the medical domain presents a unique, interdisciplinary challenge, combining fields such as Computer Vision, Natural Language Processing, and Knowledge Representation. Despite its importance, research in medical VQA has been scant, only gaining momentum since 2018. Addressing this gap, our research delves into the effective representation of radiology images and the joint learning of multimodal representations, surpassing existing methods. We innovatively augment the SLAKE dataset, enabling our model to respond to a more diverse array of questions, not limited to the immediate content of radiology or pathology images. Our model achieves a top-1 accuracy of 79.55\% with a less complex architecture, demonstrating comparable performance to current state-of-the-art models. This research not only advances medical VQA but also opens avenues for practical applications in diagnostic settings.

Free Form Medical Visual Question Answering in Radiology

TL;DR

The paper tackles free-form medical VQA in radiology by addressing data scarcity with targeted dataset augmentation and by leveraging intra-domain transfer learning through radiology-pretrained image encoders. It introduces MedCLiP-based cross-modal supervision to learn robust multi-modal representations and frames VQA as a classification task to streamline benchmarking on SLAKE. Empirical results show intra-domain pretraining with radiology data yields strong performance, with the MedCLiP approach achieving a top-1 accuracy of 79.55% on SLAKE, comparable to state-of-the-art while using a simpler architecture. The work highlights practical implications for diagnostic-support tools, though it notes limitations in explainability and vocabulary scope that warrant future improvements in interpretability and open-ended responses.

Abstract

Visual Question Answering (VQA) in the medical domain presents a unique, interdisciplinary challenge, combining fields such as Computer Vision, Natural Language Processing, and Knowledge Representation. Despite its importance, research in medical VQA has been scant, only gaining momentum since 2018. Addressing this gap, our research delves into the effective representation of radiology images and the joint learning of multimodal representations, surpassing existing methods. We innovatively augment the SLAKE dataset, enabling our model to respond to a more diverse array of questions, not limited to the immediate content of radiology or pathology images. Our model achieves a top-1 accuracy of 79.55\% with a less complex architecture, demonstrating comparable performance to current state-of-the-art models. This research not only advances medical VQA but also opens avenues for practical applications in diagnostic settings.
Paper Structure (13 sections, 4 figures, 1 table)

This paper contains 13 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Literature Survey
  3. Existing Datasets
  4. Related Work
  5. Proposed Methodology
  6. Baseline models
  7. Data Augmentation
  8. Image Encoder Pre-training
  9. Joint learning of effective multi-modal representations with cross-modal supervision
  10. Data And Experiment Setup
  11. Results
  12. Discussion
  13. Limitations

Figures (4)

  • Figure 1: Joint embedding framework for Medical VQA survey
  • Figure 2: Predictions of proposed model vs that of PubMedClip pubMedClip
  • Figure 3: Training and validation learning curves for our experiments
  • Figure 4: Performance on questions not in the SLAKE dataset