Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Efficient and Robust VQA-NLE Data Generation with Large Vision-Language Models

Patrick Amadeus Irawan, Genta Indra Winata, Samuel Cahyawijaya, Ayu Purwarianti

TL;DR

This work tackles the data bottleneck in Vision Question Answering with Natural Language Explanations (VQA-NLE) by leveraging large vision-language models (LVLMs) to generate synthetic, high-quality VQA-NLE data. It proposes three prompting pipelines—Single-Step, Single-Step-ViP, and Multi-Step—to produce triplets (question, answer, explanation) from images, with visual prompts improving relevancy. The authors evaluate data quality and distributional similarity against a human-annotated baseline on a 10k-GQA-derived dataset, showing up to $20\times$ efficiency gain with minimal accuracy loss (near-human quality). They also analyze the impact of model size and visual prompting on instruction obedience and data quality, and discuss limitations and ethical considerations. The work demonstrates a scalable and robust approach to automated multi-modal NLE data generation that reduces reliance on costly human labeling.

Abstract

Natural Language Explanation (NLE) aims to elucidate the decision-making process by providing detailed, human-friendly explanations in natural language. It helps demystify the decision-making processes of large vision-language models (LVLMs) through the use of language models. While existing methods for creating a Vision Question-Answering with Natural Language Explanation (VQA-NLE) datasets can provide explanations, they heavily rely on human annotations that are time-consuming and costly. In this study, we propose a novel approach that leverages LVLMs to efficiently generate high-quality synthetic VQA-NLE datasets. By evaluating our synthetic data, we showcase how advanced prompting techniques can lead to the production of high-quality VQA-NLE data. Our findings indicate that this proposed method achieves up to 20x faster than human annotation, with only a minimal decrease in qualitative metrics, achieving robust quality that is nearly equivalent to human-annotated data. Furthermore, we show that incorporating visual prompts significantly enhances the relevance of text generation. Our study paves the way for a more efficient and robust automated generation of multi-modal NLE data, offering a promising solution to the problem.

Towards Efficient and Robust VQA-NLE Data Generation with Large Vision-Language Models

TL;DR

This work tackles the data bottleneck in Vision Question Answering with Natural Language Explanations (VQA-NLE) by leveraging large vision-language models (LVLMs) to generate synthetic, high-quality VQA-NLE data. It proposes three prompting pipelines—Single-Step, Single-Step-ViP, and Multi-Step—to produce triplets (question, answer, explanation) from images, with visual prompts improving relevancy. The authors evaluate data quality and distributional similarity against a human-annotated baseline on a 10k-GQA-derived dataset, showing up to efficiency gain with minimal accuracy loss (near-human quality). They also analyze the impact of model size and visual prompting on instruction obedience and data quality, and discuss limitations and ethical considerations. The work demonstrates a scalable and robust approach to automated multi-modal NLE data generation that reduces reliance on costly human labeling.

Abstract

Natural Language Explanation (NLE) aims to elucidate the decision-making process by providing detailed, human-friendly explanations in natural language. It helps demystify the decision-making processes of large vision-language models (LVLMs) through the use of language models. While existing methods for creating a Vision Question-Answering with Natural Language Explanation (VQA-NLE) datasets can provide explanations, they heavily rely on human annotations that are time-consuming and costly. In this study, we propose a novel approach that leverages LVLMs to efficiently generate high-quality synthetic VQA-NLE datasets. By evaluating our synthetic data, we showcase how advanced prompting techniques can lead to the production of high-quality VQA-NLE data. Our findings indicate that this proposed method achieves up to 20x faster than human annotation, with only a minimal decrease in qualitative metrics, achieving robust quality that is nearly equivalent to human-annotated data. Furthermore, we show that incorporating visual prompts significantly enhances the relevance of text generation. Our study paves the way for a more efficient and robust automated generation of multi-modal NLE data, offering a promising solution to the problem.
Paper Structure (38 sections, 4 equations, 3 figures, 13 tables, 3 algorithms)

This paper contains 38 sections, 4 equations, 3 figures, 13 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Methodology
  3. Single-Step
  4. Single-Step-ViP
  5. Multi-Step
  6. Experimental Setup
  7. Dataset
  8. Models
  9. Prompts
  10. Experiment Settings
  11. Evaluation Settings
  12. Quality Evaluation
  13. Similarity Evaluation
  14. Validity and Efficiency Evaluation
  15. Results and Analysis
  16. ...and 23 more sections

Figures (3)

  • Figure 1: Generated VQA data along with NLE of the predicted answers, offering better explainability over traditional VQA data. These are the three samples from our synthetic VQA-NLE dataset. We create a total of 66,682 unique instances of these triplets.
  • Figure 2: An illustrative example of how we construct $\mathcal{T}(q,a,e)$ with three different approaches ($\textcolor{black}{Single-Step}$, $\textcolor{black}{Single-Step-ViP}$, and $\textcolor{black}{Multi-Step}$) using model $M$, prompt $p$, and image $D$. Each $D_i$ is used to generate up to $j$ triplets using different formatted prompt $p_{j}$ with supporting instruction (e.g., question prefix).
  • Figure 3: Comparison of density estimation for sentence length distribution across all experiment settings and human-generated $\mathcal{T}$. It provides a visual representation of the distribution differences, with more detailed numerical insights available in Table \ref{['tab:similarity_table']}.