Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SimpsonsVQA: Enhancing Inquiry-Based Learning with a Tailored Dataset

Ngoc Dung Huynh, Mohamed Reda Bouadjenek, Sunil Aryal, Imran Razzak, Hakim Hacid

TL;DR

SimpsonsVQA is a novel dataset for VQA derived from The Simpsons TV show, designed to promote inquiry-based learning and show that current large vision-language models like ChatGPT4o underperform in zero-shot settings across all three tasks, highlighting the dataset's value for improving model performance on cartoon images.

Abstract

Visual Question Answering (VQA) has emerged as a promising area of research to develop AI-based systems for enabling interactive and immersive learning. Numerous VQA datasets have been introduced to facilitate various tasks, such as answering questions or identifying unanswerable ones. However, most of these datasets are constructed using real-world images, leaving the performance of existing models on cartoon images largely unexplored. Hence, in this paper, we present "SimpsonsVQA", a novel dataset for VQA derived from The Simpsons TV show, designed to promote inquiry-based learning. Our dataset is specifically designed to address not only the traditional VQA task but also to identify irrelevant questions related to images, as well as the reverse scenario where a user provides an answer to a question that the system must evaluate (e.g., as correct, incorrect, or ambiguous). It aims to cater to various visual applications, harnessing the visual content of "The Simpsons" to create engaging and informative interactive systems. SimpsonsVQA contains approximately 23K images, 166K QA pairs, and 500K judgments (https://simpsonsvqa.org). Our experiments show that current large vision-language models like ChatGPT4o underperform in zero-shot settings across all three tasks, highlighting the dataset's value for improving model performance on cartoon images. We anticipate that SimpsonsVQA will inspire further research, innovation, and advancements in inquiry-based learning VQA.

SimpsonsVQA: Enhancing Inquiry-Based Learning with a Tailored Dataset

TL;DR

SimpsonsVQA is a novel dataset for VQA derived from The Simpsons TV show, designed to promote inquiry-based learning and show that current large vision-language models like ChatGPT4o underperform in zero-shot settings across all three tasks, highlighting the dataset's value for improving model performance on cartoon images.

Abstract

Visual Question Answering (VQA) has emerged as a promising area of research to develop AI-based systems for enabling interactive and immersive learning. Numerous VQA datasets have been introduced to facilitate various tasks, such as answering questions or identifying unanswerable ones. However, most of these datasets are constructed using real-world images, leaving the performance of existing models on cartoon images largely unexplored. Hence, in this paper, we present "SimpsonsVQA", a novel dataset for VQA derived from The Simpsons TV show, designed to promote inquiry-based learning. Our dataset is specifically designed to address not only the traditional VQA task but also to identify irrelevant questions related to images, as well as the reverse scenario where a user provides an answer to a question that the system must evaluate (e.g., as correct, incorrect, or ambiguous). It aims to cater to various visual applications, harnessing the visual content of "The Simpsons" to create engaging and informative interactive systems. SimpsonsVQA contains approximately 23K images, 166K QA pairs, and 500K judgments (https://simpsonsvqa.org). Our experiments show that current large vision-language models like ChatGPT4o underperform in zero-shot settings across all three tasks, highlighting the dataset's value for improving model performance on cartoon images. We anticipate that SimpsonsVQA will inspire further research, innovation, and advancements in inquiry-based learning VQA.

Paper Structure

This paper contains 15 sections, 11 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. SimpsonsVQA Dataset
  4. Dataset Creation
  5. Task Description
  6. SimpsonsVQA Dataset Analysis
  7. Question Analysis
  8. Answer Analysis
  9. Experimental Evaluation
  10. Results of the Conventional VQA Task
  11. Results of the Question Relevance Task
  12. Results of the Answer Correctness Task
  13. Discussion
  14. Potential Negative Societal Impact
  15. Conclusion & Future Work

Figures (11)

  • Figure 1: Examples from the SimpsonsVQA dataset.
  • Figure 2: AMT assessment interface.
  • Figure 3: Ratio of question relevance as judged by AMT workers.
  • Figure 4: Distribution of first words in questions. The words are arranged in an outward radiating pattern from the center, with their ordering based on frequency. The size of the arcs corresponds to the number of questions containing each word.
  • Figure 5: Question topics.
  • ...and 6 more figures