Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RelationalFactQA: A Benchmark for Evaluating Tabular Fact Retrieval from Large Language Models

Dario Satriani, Enzo Veltri, Donatello Santoro, Paolo Papotti

TL;DR

This paper introduces RelationalFactQA, a benchmark for evaluating LLMs' ability to generate relational tabular outputs from internal knowledge without external tools. It demonstrates that structured multi-record factual retrieval is substantially harder than single facts, with models rarely exceeding 25% tuple-level accuracy and performance deteriorating as output size grows. The authors propose a hybrid RFQA dataset (696 NL/SQL triples across nine domains) and three retrieval modes (NL, SQL, CoT), along with F1 and Tuple Similarity metrics, to systematically study this capability. The findings highlight critical limitations in current LLMs' capacity to synthesize structured factual knowledge, and position RelationalFactQA as a key resource for future factuality research and progress tracking.

Abstract

Factuality in Large Language Models (LLMs) is a persistent challenge. Current benchmarks often assess short factual answers, overlooking the critical ability to generate structured, multi-record tabular outputs from parametric knowledge. We demonstrate that this relational fact retrieval is substantially more difficult than isolated point-wise queries, even when individual facts are known to the model, exposing distinct failure modes sensitive to output dimensionality (e.g., number of attributes or records). To systematically evaluate this under-explored capability, we introduce RelationalFactQA, a new benchmark featuring diverse natural language questions (paired with SQL) and gold-standard tabular answers, specifically designed to assess knowledge retrieval in a structured format. RelationalFactQA enables analysis across varying query complexities, output sizes, and data characteristics. Our experiments reveal that even state-of-the-art LLMs struggle significantly, not exceeding 25% factual accuracy in generating relational outputs, with performance notably degrading as output dimensionality increases. These findings underscore critical limitations in current LLMs' ability to synthesize structured factual knowledge and establish RelationalFactQA as a crucial resource for measuring future progress in LLM factuality.

RelationalFactQA: A Benchmark for Evaluating Tabular Fact Retrieval from Large Language Models

TL;DR

This paper introduces RelationalFactQA, a benchmark for evaluating LLMs' ability to generate relational tabular outputs from internal knowledge without external tools. It demonstrates that structured multi-record factual retrieval is substantially harder than single facts, with models rarely exceeding 25% tuple-level accuracy and performance deteriorating as output size grows. The authors propose a hybrid RFQA dataset (696 NL/SQL triples across nine domains) and three retrieval modes (NL, SQL, CoT), along with F1 and Tuple Similarity metrics, to systematically study this capability. The findings highlight critical limitations in current LLMs' capacity to synthesize structured factual knowledge, and position RelationalFactQA as a key resource for future factuality research and progress tracking.

Abstract

Factuality in Large Language Models (LLMs) is a persistent challenge. Current benchmarks often assess short factual answers, overlooking the critical ability to generate structured, multi-record tabular outputs from parametric knowledge. We demonstrate that this relational fact retrieval is substantially more difficult than isolated point-wise queries, even when individual facts are known to the model, exposing distinct failure modes sensitive to output dimensionality (e.g., number of attributes or records). To systematically evaluate this under-explored capability, we introduce RelationalFactQA, a new benchmark featuring diverse natural language questions (paired with SQL) and gold-standard tabular answers, specifically designed to assess knowledge retrieval in a structured format. RelationalFactQA enables analysis across varying query complexities, output sizes, and data characteristics. Our experiments reveal that even state-of-the-art LLMs struggle significantly, not exceeding 25% factual accuracy in generating relational outputs, with performance notably degrading as output dimensionality increases. These findings underscore critical limitations in current LLMs' ability to synthesize structured factual knowledge and establish RelationalFactQA as a crucial resource for measuring future progress in LLM factuality.

Paper Structure

This paper contains 19 sections, 21 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Factuality Evaluation Benchmarks.
  4. Table Question Answering and Reasoning.
  5. Text-to-SQL.
  6. Knowledge Probes for LLMs.
  7. The Benchmark
  8. Experimental Settings
  9. Results
  10. Conclusions
  11. Motivation Example
  12. Experiment 1: Incremental Attribute Request.
  13. Experiment 2: Attribute Position Sensitivity.
  14. More Details on RelationalFactQA Dataset
  15. Prompts and LLM Response Processing
  16. ...and 4 more sections

Figures (21)

  • Figure 2: TS results for LLama 3.3, GPT-4.1 and QWEN 3, with all retrieval techniques, w.r.t. the expected output measure as the number of attributes (left) and cells (right).
  • Figure 4: Topic distribution
  • Figure 5: NL Prompt Syntax. Text in italic is injected from the given NL query and the expected JSON schema of the response.
  • Figure 6: SQL Prompt Syntax. Text in italic is injected from the given SQL query and the expected JSON schema of the response.
  • Figure 7: CoT Prompt Syntax. Text in italic is injected from the given SQL query. Values between parenthesis are populated only if the condition(s) is given.
  • ...and 16 more figures