Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

StackOverflowVQA: Stack Overflow Visual Question Answering Dataset

Motahhare Mirzaei, Mohammad Javad Pirhadi, Sauleh Eetemadi

TL;DR

StackOverflowVQA tackles the gap in software-focused VQA by compiling StackOverflow questions that include images and yield full-sentence human answers. The authors introduce the StackOverflowVQA and its filtered variants, and evaluate a GIT-based baseline to assess feasibility of image-guided programming QA. They provide dataset statistics, analyze when images are essential for answering, and discuss evaluation challenges for this domain. The work paves the way for visually aware programming assistants and releases the data for public use on HuggingFace, highlighting domain-specific hurdles and potential improvements.

Abstract

In recent years, people have increasingly used AI to help them with their problems by asking questions on different topics. One of these topics can be software-related and programming questions. In this work, we focus on the questions which need the understanding of images in addition to the question itself. We introduce the StackOverflowVQA dataset, which includes questions from StackOverflow that have one or more accompanying images. This is the first VQA dataset that focuses on software-related questions and contains multiple human-generated full-sentence answers. Additionally, we provide a baseline for answering the questions with respect to images in the introduced dataset using the GIT model. All versions of the dataset are available at https://huggingface.co/mirzaei2114.

StackOverflowVQA: Stack Overflow Visual Question Answering Dataset

TL;DR

StackOverflowVQA tackles the gap in software-focused VQA by compiling StackOverflow questions that include images and yield full-sentence human answers. The authors introduce the StackOverflowVQA and its filtered variants, and evaluate a GIT-based baseline to assess feasibility of image-guided programming QA. They provide dataset statistics, analyze when images are essential for answering, and discuss evaluation challenges for this domain. The work paves the way for visually aware programming assistants and releases the data for public use on HuggingFace, highlighting domain-specific hurdles and potential improvements.

Abstract

In recent years, people have increasingly used AI to help them with their problems by asking questions on different topics. One of these topics can be software-related and programming questions. In this work, we focus on the questions which need the understanding of images in addition to the question itself. We introduce the StackOverflowVQA dataset, which includes questions from StackOverflow that have one or more accompanying images. This is the first VQA dataset that focuses on software-related questions and contains multiple human-generated full-sentence answers. Additionally, we provide a baseline for answering the questions with respect to images in the introduced dataset using the GIT model. All versions of the dataset are available at https://huggingface.co/mirzaei2114.
Paper Structure (10 sections, 3 figures, 2 tables)

This paper contains 10 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Dataset
  3. StackOverflowVQA
  4. StackOverflowVQA-filtered
  5. StackOverflowVQA-filtered-small
  6. Model
  7. Training
  8. Results
  9. Future Work
  10. Conclusion

Figures (3)

  • Figure 1: Distribution of images with respect to relatedness and requirement. (RR=Related and required, R=Related but not required, U=Unrelated)
  • Figure 2: Examples of a required image (left) and a related image that is not required to answer the question (right).
  • Figure 3: Frequency of questions based on answer counts (left) and image counts (right) in StackOverflowVQA dataset