Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SynClaimEval: A Framework for Evaluating the Utility of Synthetic Data in Long-Context Claim Verification

Mohamed Elaraby, Jyoti Prakash Maheswari

TL;DR

The paper introduces SynClaimEval, a framework to systematically evaluate the usefulness of synthetic data for long-context claim verification across three dimensions: training context length, synthesis logic, and explanation quality.It shows that longer training contexts generally improve verification, and that structured synthesis strategies, especially argument-graph-based approaches, yield the strongest gains for certain models while also enhancing explanation quality.The work demonstrates that synthetic data can serve as a scalable augmentation to human-annotated data, improving both predictive performance and the grounding of model explanations, with notable benefits when combined with existing datasets.Overall, SynClaimEval provides a principled methodology and empirical evidence for leveraging synthetic data to advance long-context reasoning and interpretability in large language models.

Abstract

Large Language Models (LLMs) with extended context windows promise direct reasoning over long documents, reducing the need for chunking or retrieval. Constructing annotated resources for training and evaluation, however, remains costly. Synthetic data offers a scalable alternative, and we introduce SynClaimEval, a framework for evaluating synthetic data utility in long-context claim verification -- a task central to hallucination detection and fact-checking. Our framework examines three dimensions: (i) input characteristics, by varying context length and testing generalization to out-of-domain benchmarks; (ii) synthesis logic, by controlling claim complexity and error type variation; and (iii) explanation quality, measuring the degree to which model explanations provide evidence consistent with predictions. Experiments across benchmarks show that long-context synthesis can improve verification in base instruction-tuned models, particularly when augmenting existing human-written datasets. Moreover, synthesis enhances explanation quality, even when verification scores do not improve, underscoring its potential to strengthen both performance and explainability.

SynClaimEval: A Framework for Evaluating the Utility of Synthetic Data in Long-Context Claim Verification

TL;DR

The paper introduces SynClaimEval, a framework to systematically evaluate the usefulness of synthetic data for long-context claim verification across three dimensions: training context length, synthesis logic, and explanation quality.It shows that longer training contexts generally improve verification, and that structured synthesis strategies, especially argument-graph-based approaches, yield the strongest gains for certain models while also enhancing explanation quality.The work demonstrates that synthetic data can serve as a scalable augmentation to human-annotated data, improving both predictive performance and the grounding of model explanations, with notable benefits when combined with existing datasets.Overall, SynClaimEval provides a principled methodology and empirical evidence for leveraging synthetic data to advance long-context reasoning and interpretability in large language models.

Abstract

Large Language Models (LLMs) with extended context windows promise direct reasoning over long documents, reducing the need for chunking or retrieval. Constructing annotated resources for training and evaluation, however, remains costly. Synthetic data offers a scalable alternative, and we introduce SynClaimEval, a framework for evaluating synthetic data utility in long-context claim verification -- a task central to hallucination detection and fact-checking. Our framework examines three dimensions: (i) input characteristics, by varying context length and testing generalization to out-of-domain benchmarks; (ii) synthesis logic, by controlling claim complexity and error type variation; and (iii) explanation quality, measuring the degree to which model explanations provide evidence consistent with predictions. Experiments across benchmarks show that long-context synthesis can improve verification in base instruction-tuned models, particularly when augmenting existing human-written datasets. Moreover, synthesis enhances explanation quality, even when verification scores do not improve, underscoring its potential to strengthen both performance and explainability.

Paper Structure

This paper contains 26 sections, 1 equation, 4 figures, 16 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. SynClaimEval
  4. Preparing Claim Sources
  5. Argument-graph Synthesis.
  6. Evaluating Explanations (RQ3)
  7. Datasets
  8. Synthetic Sources
  9. Evaluation Benchmarks
  10. Experimental Setup
  11. Models and Prompting
  12. Continual Fine-tuning
  13. Results and Analysis
  14. RQ1: Context Length and Domain Generalization
  15. RQ2: Error types and synthesis logic
  16. ...and 11 more sections

Figures (4)

  • Figure 1: Overview of the SynClaimEval pipeline. The framework is designed to evaluate synthetic data along three dimensions: (1) context length and domain effects, (2) claim generation logic, and (3) explanation quality.
  • Figure 2: Context length effect on scoring
  • Figure 3: Error types effect
  • Figure 4: Pairwise supportiveness ranking of explanations across benchmarks. Colors denote synthesis type (Base, Unstructured, Context-graph, Argument-graph). Higher scores indicate stronger judged quality.