Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Instruction Makes a Difference

Tosin Adewumi, Nudrat Habib, Lama Alkhaled, Elisa Barney

TL;DR

Problem addressed: document VQA requires answers from document images; Approach: introduce iDocVQA and LLaDoc built on LLaVA1.5-7B to enable instruction-following document analysis; Method: evaluate four training regimes (zero-shot, finetuning, instruction-tuning, 50-50) on merged idvqa across docvqa, tvqa, idvqa, with POPE hallucination testing; Findings: instruction-tuning yields substantial gains, from $11\times$ to $32\times$ improvement over zero-shot and $0.1\%$ to $4.2\%$ gains over non-instruction finetuning, though human performance remains at $94.36\%$; Contributions: public release of iDocVQA dataset and the llad model, demonstrating the value of instruction-following in document understanding and outlining future directions like multilingual data and multi-image processing.

Abstract

We introduce Instruction Document Visual Question Answering (iDocVQA) dataset and Large Language Document (LLaDoc) model, for training Language-Vision (LV) models for document analysis and predictions on document images, respectively. Usually, deep neural networks for the DocVQA task are trained on datasets lacking instructions. We show that using instruction-following datasets improves performance. We compare performance across document-related datasets using the recent state-of-the-art (SotA) Large Language and Vision Assistant (LLaVA)1.5 as the base model. We also evaluate the performance of the derived models for object hallucination using the Polling-based Object Probing Evaluation (POPE) dataset. The results show that instruction-tuning performance ranges from 11X to 32X of zero-shot performance and from 0.1% to 4.2% over non-instruction (traditional task) finetuning. Despite the gains, these still fall short of human performance (94.36%), implying there's much room for improvement.

Instruction Makes a Difference

TL;DR

Problem addressed: document VQA requires answers from document images; Approach: introduce iDocVQA and LLaDoc built on LLaVA1.5-7B to enable instruction-following document analysis; Method: evaluate four training regimes (zero-shot, finetuning, instruction-tuning, 50-50) on merged idvqa across docvqa, tvqa, idvqa, with POPE hallucination testing; Findings: instruction-tuning yields substantial gains, from to improvement over zero-shot and to gains over non-instruction finetuning, though human performance remains at ; Contributions: public release of iDocVQA dataset and the llad model, demonstrating the value of instruction-following in document understanding and outlining future directions like multilingual data and multi-image processing.

Abstract

We introduce Instruction Document Visual Question Answering (iDocVQA) dataset and Large Language Document (LLaDoc) model, for training Language-Vision (LV) models for document analysis and predictions on document images, respectively. Usually, deep neural networks for the DocVQA task are trained on datasets lacking instructions. We show that using instruction-following datasets improves performance. We compare performance across document-related datasets using the recent state-of-the-art (SotA) Large Language and Vision Assistant (LLaVA)1.5 as the base model. We also evaluate the performance of the derived models for object hallucination using the Polling-based Object Probing Evaluation (POPE) dataset. The results show that instruction-tuning performance ranges from 11X to 32X of zero-shot performance and from 0.1% to 4.2% over non-instruction (traditional task) finetuning. Despite the gains, these still fall short of human performance (94.36%), implying there's much room for improvement.
Paper Structure (7 sections, 9 figures, 4 tables)

This paper contains 7 sections, 9 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Methodology
  4. idvqa Dataset Creation
  5. Results and Discussion
  6. Conclusion
  7. Limitations

Figures (9)

  • Figure 1: llad architecture/training schema
  • Figure 2: Spider chart of the performance of the models.
  • Figure 3: docvqa example where Instruction-tuning outperforms others. Q: What is the brand name of the chips/snacks produced by ITC?" Correct: Instruction-tuning: Bingo, Incorrect: Finetuning: Tangles
  • Figure 4: docvqa example where Instruction-tuning outperforms others. Q: What is the variable taken along the x axis ? Correct: weeks of consumption, Incorrect: Finetuning: weeks
  • Figure 5: tvqa example where Instruction-tuning outperforms others. Q: what is the largest measurement we can see on this ruler? Correct: Instruction-tuning: 50, Incorrect: Finetuning: 40
  • ...and 4 more figures