KG-MuLQA: A Framework for KG-based Multi-Level QA Extraction and Long-Context LLM Evaluation

TL;DR

KG-MuLQA introduces a knowledge-graph–driven framework to extract multi-level QA pairs from annotated documents and to evaluate long-context LLMs with controlled complexity. By converting 170 SEC credit agreements into RDF-based knowledge graphs and generating QA templates across five complexity levels via SPARQL queries, the authors create KG-MuLQA-D with 20,139 QA pairs. The evaluation spans 16 proprietary and open-weight LLMs, revealing systematic weaknesses in multi-hop reasoning, set-based operations, and implicit-relations grounding, even for strong models. The work provides a scalable, interpretable benchmark, releases data and tooling, and demonstrates applicability to domains beyond finance, including medical documents, while outlining human performance baselines and actionable insights for improving long-context QA systems.

Abstract

We introduce KG-MuLQA (Knowledge-Graph-based Multi-Level Question-Answer Extraction): a framework that (1) extracts QA pairs at multiple complexity levels (2) along three key dimensions -- multi-hop retrieval, set operations, and answer plurality, (3) by leveraging knowledge-graph-based document representations. This approach enables fine-grained assessment of model performance across controlled difficulty levels. Using this framework, we construct a dataset of 20,139 QA pairs based on financial credit agreements and evaluate 16 proprietary and open-weight Large Language Models, observing that even the best-performing models struggle with set-based comparisons and multi-hop reasoning over long contexts. Our analysis reveals systematic failure modes tied to semantic misinterpretation and inability to handle implicit relations.

Figures (12)

• Figure 1: Overview of KG-MuLQA. Credit agreements are annotated to identify entities and their relationships, forming a knowledge graph representation. This graph is then used to systematically extract multi-level QA pairs, which serve as the basis for benchmarking long-context LLMs.
• Figure 2: Knowledge graph representation of annotated credit agreements. The ontology layer defines high-level entity types and their sub-classes (e.g., Agent, Lender), while the data layer contains document-specific instances connected via labeled relations (e.g., hasEmployee, isLocationOf). This abstraction supports systematic QA extraction across documents by removing company-specific constraints. All entity names in this figure, including individuals, organizations, and addresses, are fictional and used for illustrative purposes only.
• Figure 3: This figure illustrates the multi-stage process used to evaluate long-context LLMs: documents are split into chunks (if exceeding 128K tokens), paired with question batches ($\leqslant$50), and sequentially fed to the model. Model responses are collected per chunk and per question batch to ensure evaluation scalability across large documents.
• Figure 4: Trends in error types across complexity levels. The template level is given by $P + H + \#SO$.
• Figure 5: Representative Annotation Scenarios from Label Studio.