Layout and Task Aware Instruction Prompt for Zero-shot Document Image Question Answering

TL;DR

The paper challenges the need for layout-centered pre-training in document image QA by showing that instruction-tuning LMs can interpret document layout through spaces and line breaks. It introduces LATIN-Prompt, which converts OCR outputs into a layout-aware text stream and enforces task-specific output formats, enabling zero-shot QA with Claude and ChatGPT that rivals fine-tuned SOTA methods. To empower smaller models, LATIN-Tuning crafts instruction-tuning data from CSV tables to substantially boost Alpaca’s DocVQA performance. Across DocVQA, InfographicVQA, and MP-DocVQA, LATIN-Prompt delivers large zero-shot gains, while LATIN-Tuning narrows the gap for smaller models, marking a practical path toward scalable, layout-aware document understanding without heavy pre-training.

Abstract

Layout-aware pre-trained models has achieved significant progress on document image question answering. They introduce extra learnable modules into existing language models to capture layout information within document images from text bounding box coordinates obtained by OCR tools. However, extra modules necessitate pre-training on extensive document images. This prevents these methods from directly utilizing off-the-shelf instruction-tuning language foundation models, which have recently shown promising potential in zero-shot learning. Instead, in this paper, we find that instruction-tuning language models like Claude and ChatGPT can understand layout by spaces and line breaks. Based on this observation, we propose the LAyout and Task aware Instruction Prompt (LATIN-Prompt), which consists of layout-aware document content and task-aware instruction. Specifically, the former uses appropriate spaces and line breaks to recover the layout information among text segments obtained by OCR tools, and the latter ensures that generated answers adhere to formatting requirements. Moreover, we propose the LAyout and Task aware Instruction Tuning (LATIN-Tuning) to improve the performance of small instruction-tuning models like Alpaca. Experimental results show that LATIN-Prompt enables zero-shot performance of Claude and ChatGPT to be comparable to the fine-tuning performance of SOTAs on document image question answering, and LATIN-Tuning enhances the zero-shot performance of Alpaca significantly. For example, LATIN-Prompt improves the performance of Claude and ChatGPT on DocVQA by 263% and 20% respectively. LATIN-Tuning improves the performance of Alpaca on DocVQA by 87.7%. Quantitative and qualitative analyses demonstrate the effectiveness of LATIN-Prompt and LATIN-Tuning. We provide the code in supplementary and will release it to facilitate future research.

Figures (6)

• Figure 1: (a) Existing methods introduce layout-aware modules into language models to capture layout information within document images from text bounding box coordinates obtained by OCR tools. They need further pre-training on extensive document images. (b) Our method allows instruction-tuning language models to capture layout by spaces and line breaks and can conduct zero-shot inference on document image question-answering.
• Figure 2: The overview of LATIN-Prompt (\ref{['sec:LATIN']}). Given a document image and the corresponding question, we recover the layout information within the document image from OCR results using appropriate spaces and line breaks, and then insert the layout aware document content and question into the task instruction prompt template together. The instruction-tuning large language foundation model takes the filled template as input and predicts the answer to the question in the required format.
• Figure 3: Construction of LATIN-Tuning data (\ref{['sec:latin-tuning']}). (1) Convert the CSV-format table into document string with spaces and line breaks by Pandas. (2) Insert document string into the Question Generation Prompt Template and generate a question-answer pair by Claude. (3) Insert document string and question into the Instruction Prompt Template to form the input, with the answer serving as the target.
• Figure 4: The impact of the size of instruction fine-tuning dataset on LATIN-Tuning. The performance of LATIN-Tuning improves as the number of samples increases.
• Figure 5: Case study of Claude on DocVQA. Due to the lack of task instruction, Plain Prompt generates unnecessary words (in blue), violating the extraction requirement. Moreover, Plain Prompt cannot capture layout information and generates incorrect answers (in green). In Case (A), it regards "Coffee" and "Chocolate", which are semantically similar to "choco", as the answer. In Case (B), it regards "61" as the answer since it directly follows "romance and mystery". In Case (C), it fails to comprehend the document and engages in erroneous reasoning. In contrast, LATIN-Prompt can understand layout relationships and generate the correct answer (in red). The document images of these cases are complex. Due to limited space, we only display a portion of the original document images here. Please refer to the appendix for the original document images and more cases.