Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ERVQA: A Dataset to Benchmark the Readiness of Large Vision Language Models in Hospital Environments

Sourjyadip Ray, Kushal Gupta, Soumi Kundu, Payal Arvind Kasat, Somak Aditya, Pawan Goyal

TL;DR

This work examines the healthcare knowledge of existing Large Vision Language Models via the Visual Question Answering (VQA) task in hospital settings through expert annotated open-ended questions, and introduces the Emergency Room Visual Question Answering (ERVQA) dataset, a seminal benchmark for LVLMs.

Abstract

The global shortage of healthcare workers has demanded the development of smart healthcare assistants, which can help monitor and alert healthcare workers when necessary. We examine the healthcare knowledge of existing Large Vision Language Models (LVLMs) via the Visual Question Answering (VQA) task in hospital settings through expert annotated open-ended questions. We introduce the Emergency Room Visual Question Answering (ERVQA) dataset, consisting of <image, question, answer> triplets covering diverse emergency room scenarios, a seminal benchmark for LVLMs. By developing a detailed error taxonomy and analyzing answer trends, we reveal the nuanced nature of the task. We benchmark state-of-the-art open-source and closed LVLMs using traditional and adapted VQA metrics: Entailment Score and CLIPScore Confidence. Analyzing errors across models, we infer trends based on properties like decoder type, model size, and in-context examples. Our findings suggest the ERVQA dataset presents a highly complex task, highlighting the need for specialized, domain-specific solutions.

ERVQA: A Dataset to Benchmark the Readiness of Large Vision Language Models in Hospital Environments

TL;DR

This work examines the healthcare knowledge of existing Large Vision Language Models via the Visual Question Answering (VQA) task in hospital settings through expert annotated open-ended questions, and introduces the Emergency Room Visual Question Answering (ERVQA) dataset, a seminal benchmark for LVLMs.

Abstract

The global shortage of healthcare workers has demanded the development of smart healthcare assistants, which can help monitor and alert healthcare workers when necessary. We examine the healthcare knowledge of existing Large Vision Language Models (LVLMs) via the Visual Question Answering (VQA) task in hospital settings through expert annotated open-ended questions. We introduce the Emergency Room Visual Question Answering (ERVQA) dataset, consisting of <image, question, answer> triplets covering diverse emergency room scenarios, a seminal benchmark for LVLMs. By developing a detailed error taxonomy and analyzing answer trends, we reveal the nuanced nature of the task. We benchmark state-of-the-art open-source and closed LVLMs using traditional and adapted VQA metrics: Entailment Score and CLIPScore Confidence. Analyzing errors across models, we infer trends based on properties like decoder type, model size, and in-context examples. Our findings suggest the ERVQA dataset presents a highly complex task, highlighting the need for specialized, domain-specific solutions.
Paper Structure (36 sections, 2 equations, 13 figures, 3 tables)

This paper contains 36 sections, 2 equations, 13 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Visual Question Answering Datasets
  4. Medical Visual Question Answering
  5. ERVQA Dataset
  6. Image Curation
  7. Question Answer Annotation
  8. Data Augmentation
  9. Quality Check
  10. Dataset Statistics
  11. Error Analysis
  12. Experimental Setup
  13. Error Taxonomy
  14. Error Occurrence and Co-occurrence
  15. Benchmark
  16. ...and 21 more sections

Figures (13)

  • Figure 1: Example data point from the ERVQA dataset containing a manually annotated question and answer. The questions are asked from the point of view of a doctor.
  • Figure 2: a) VQA-RAD dataset lau2018dataset b) PathVQA he2020pathvqa c) Med VQA dataset ben2021overview d) SLAKE dataset liu2021slake e) Patient oriented dataset huang2023medical f) ERVQA dataset along with question categories. Annotators were asked not to limit themselves to just these categories. [ Disclaimer: All dataset images and QA pairs are taken from their respective papers.]
  • Figure 3: Distribution of error types amongst all 372 generated answers
  • Figure 4: Error Co-occurrence Statistics
  • Figure 5: Model wise percentages of error occurrence for comparison on the basis of LLM Decoder type
  • ...and 8 more figures