pdfQA: Diverse, Challenging, and Realistic Question Answering over PDFs

TL;DR

The paper tackles the lack of realistic, cross-domain benchmarks for evidence-based QA over PDFs by introducing pdfQA, a 4K dataset comprising synthetic syn-pdfQA and real-pdfQA. It employs a four-step synthetic data generation pipeline and integrates nine human-annotated PDF benchmarks, both subjected to stringent quality and difficulty filters to yield valid, challenging QA pairs. Experiments with open-source long-context QA systems reveal that performance correlates with complexity dimensions and that real-pdfQA is notably more challenging than synthetic data, highlighting gaps in current approaches for long-context PDF QA. Overall, pdfQA provides a comprehensive, end-to-end evaluation platform for parsing, retrieval, and reasoning in PDF QA, while acknowledging limitations related to external validity, filter conservativeness, and long-context methodologies.

Abstract

PDFs are the second-most used document type on the internet (after HTML). Yet, existing QA datasets commonly start from text sources or only address specific domains. In this paper, we present pdfQA, a multi-domain 2K human-annotated (real-pdfQA) and 2K synthetic dataset (syn-pdfQA) differentiating QA pairs in ten complexity dimensions (e.g., file type, source modality, source position, answer type). We apply and evaluate quality and difficulty filters on both datasets, obtaining valid and challenging QA pairs. We answer the questions with open-source LLMs, revealing existing challenges that correlate with our complexity dimensions. pdfQA presents a basis for end-to-end QA pipeline evaluation, testing diverse skill sets and local optimizations (e.g., in information retrieval or parsing).

Figures (8)

• Figure 1: Synthetic Data Generation Pipeline. The input is tuples of PDFs and the same document in a structured format (e.g., .tex or .html files). We use financial reports, research articles, books, and sustainability disclosures (1). These are processed into structured sources (2), upon which an LLM creates QA pairs with different complexity dimensions (3). We apply quality and difficulty filters to ensure valid and challenging QA pairs in pdfQA (4).
• Figure 2: Model performance on datasets in real-pdfQA. Correctness scores are between 1-5 (Figure \ref{['fig:answer_quality_prompt']}).
• Figure 3: Prompt for generating QA pairs. "sources" is replaced with a set of 5-15 sources from the underlying file, "guidelines" is replaced with one of the options in each category of source quantity, answer type, replicate or reasoning, modality, and difficulty, as listed in Figure \ref{['fig:guidelines_qa_pair_gen']}.
• Figure 4: Options for guidelines.
• Figure 5: Formality filter for QA quality. "Question" is replaced with the question under investigation, "Answer" with the answer under investigation. "Guideline" is replaced with its corresponding answer type as defined in Figure \ref{['fig:guidelines_qa_pair_gen']}.