Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Foundations of Global Consistency Checking with Noisy LLM Oracles

Paul He, Elke Kirschbaum, Shiva Kasiviswanathan

TL;DR

An adaptive divide-and-conquer algorithm that identifies minimal inconsistent subsets (MUSes) of facts and optionally computes minimal repairs through hitting-sets is proposed, offering a scalable framework for linguistic consistency verification with LLM-based evaluators.

Abstract

Ensuring that collections of natural-language facts are globally consistent is essential for tasks such as fact-checking, summarization, and knowledge base construction. While Large Language Models (LLMs) can assess the consistency of small subsets of facts, their judgments are noisy, and pairwise checks are insufficient to guarantee global coherence. We formalize this problem and show that verifying global consistency requires exponentially many oracle queries in the worst case. To make the task practical, we propose an adaptive divide-and-conquer algorithm that identifies minimal inconsistent subsets (MUSes) of facts and optionally computes minimal repairs through hitting-sets. Our approach has low-degree polynomial query complexity. Experiments with both synthetic and real LLM oracles show that our method efficiently detects and localizes inconsistencies, offering a scalable framework for linguistic consistency verification with LLM-based evaluators.

Foundations of Global Consistency Checking with Noisy LLM Oracles

TL;DR

An adaptive divide-and-conquer algorithm that identifies minimal inconsistent subsets (MUSes) of facts and optionally computes minimal repairs through hitting-sets is proposed, offering a scalable framework for linguistic consistency verification with LLM-based evaluators.

Abstract

Ensuring that collections of natural-language facts are globally consistent is essential for tasks such as fact-checking, summarization, and knowledge base construction. While Large Language Models (LLMs) can assess the consistency of small subsets of facts, their judgments are noisy, and pairwise checks are insufficient to guarantee global coherence. We formalize this problem and show that verifying global consistency requires exponentially many oracle queries in the worst case. To make the task practical, we propose an adaptive divide-and-conquer algorithm that identifies minimal inconsistent subsets (MUSes) of facts and optionally computes minimal repairs through hitting-sets. Our approach has low-degree polynomial query complexity. Experiments with both synthetic and real LLM oracles show that our method efficiently detects and localizes inconsistencies, offering a scalable framework for linguistic consistency verification with LLM-based evaluators.
Paper Structure (45 sections, 8 theorems, 15 equations, 1 figure, 2 tables, 2 algorithms)

This paper contains 45 sections, 8 theorems, 15 equations, 1 figure, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Local Consistency Verification.
  4. Global Consistency and Contradictions.
  5. Setting the Stage
  6. Problem Definition.
  7. Complexity Landscape.
  8. LLM as Noisy Subset-Consistency Oracle.
  9. Method
  10. Assumptions.
  11. MUS Extraction via Divide-and-Conquer.
  12. Greedy Repair via Hitting Set (Minimal Correction Set).
  13. Experiments
  14. Experiment Setting.
  15. Datasets
  16. ...and 30 more sections

Key Result

Theorem 4.2

Algorithm alg:main makes at most $I \cdot m \cdot (k\log N)$ oracle calls to LLM $O$.

Figures (1)

  • Figure 1: Scaling of query counts with number of facts $N$. Pairwise checking is quadratic, while QXR scales polylogarithmically

Theorems & Definitions (15)

  • Definition 4.1: Minimal Unsatisfiable Subset w.r.t. Oracle $O$
  • Theorem 4.2: Query Complexity of Algorithm \ref{['alg:main']}
  • Theorem A.1
  • proof
  • Theorem A.2
  • proof
  • Theorem A.3: Pairwise Insufficiency
  • proof
  • Theorem A.4: Soundness under Perfect Oracle
  • proof
  • ...and 5 more