Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysis

Yulong Hui, Yao Lu, Huanchen Zhang

TL;DR

A benchmark suite, namely Unstructured Document Analysis (UDA), that involves 2,965 real-world documents and 29,590 expert-annotated Q&A pairs is introduced and the design choices and answer qualities across multiple document domains and diverse query types are evaluated.

Abstract

The use of Retrieval-Augmented Generation (RAG) has improved Large Language Models (LLMs) in collaborating with external data, yet significant challenges exist in real-world scenarios. In areas such as academic literature and finance question answering, data are often found in raw text and tables in HTML or PDF formats, which can be lengthy and highly unstructured. In this paper, we introduce a benchmark suite, namely Unstructured Document Analysis (UDA), that involves 2,965 real-world documents and 29,590 expert-annotated Q&A pairs. We revisit popular LLM- and RAG-based solutions for document analysis and evaluate the design choices and answer qualities across multiple document domains and diverse query types. Our evaluation yields interesting findings and highlights the importance of data parsing and retrieval. We hope our benchmark can shed light and better serve real-world document analysis applications. The benchmark suite and code can be found at https://github.com/qinchuanhui/UDA-Benchmark.

UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysis

TL;DR

A benchmark suite, namely Unstructured Document Analysis (UDA), that involves 2,965 real-world documents and 29,590 expert-annotated Q&A pairs is introduced and the design choices and answer qualities across multiple document domains and diverse query types are evaluated.

Abstract

The use of Retrieval-Augmented Generation (RAG) has improved Large Language Models (LLMs) in collaborating with external data, yet significant challenges exist in real-world scenarios. In areas such as academic literature and finance question answering, data are often found in raw text and tables in HTML or PDF formats, which can be lengthy and highly unstructured. In this paper, we introduce a benchmark suite, namely Unstructured Document Analysis (UDA), that involves 2,965 real-world documents and 29,590 expert-annotated Q&A pairs. We revisit popular LLM- and RAG-based solutions for document analysis and evaluate the design choices and answer qualities across multiple document domains and diverse query types. Our evaluation yields interesting findings and highlights the importance of data parsing and retrieval. We hope our benchmark can shed light and better serve real-world document analysis applications. The benchmark suite and code can be found at https://github.com/qinchuanhui/UDA-Benchmark.
Paper Structure (25 sections, 4 figures, 17 tables)

This paper contains 25 sections, 4 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Retrieval Augmented Generation
  4. Dataset: UDA
  5. Dataset Composition.
  6. Label Collection.
  7. Dataset Construction.
  8. Benchmarks and Evaluations
  9. Metrics
  10. Experiment setups
  11. Evaluating Data Parsing
  12. Remark.
  13. Indexing and Retrieval
  14. RAG vs. Long Context
  15. Evaluating Chain-of-Thought and Code Interpreters
  16. ...and 10 more sections

Figures (4)

  • Figure 1: An example of basic RAG processing on unstructured documents.
  • Figure 2: The procedure of the table parsing experiment
  • Figure 3: An example of table parsing with different strategies. Raw-text-extraction preserves the informational content with structural markers; CV-based method may struggle with the irregular table presentation; GPT-4-Omni yields the highest accuracy.
  • Figure 4: Data example of FinHybrid in UDA