Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Critical Confabulation: Can LLMs Hallucinate for Social Good?

Peiqi Sui, Eamon Duede, Hoyt Long, Richard Jean So

TL;DR

The paper addresses archival gaps caused by social inequality and proposes critical confabulation as a bounded narrative cloze framework that uses LLMs to reconstruct evidence-bound missing events in historical timelines. It grounds its evaluation in the BWTC dataset, creating ground-truth timelines for hidden figures and validating them through automatic perturbations and human annotation. Experimental results show that prompt design, especially event-type cues, can elicit meaningful confabulations, with GPT-5-chat achieving approaching 60% accuracy under strong prompting, while open-weight models also contribute substantially. The work highlights the potential of AI-assisted humanities research to surface latent histories while underscoring the need for robust evaluation, provenance tracking, and ethical safeguards as the approach scales across languages and archives.

Abstract

LLMs hallucinate, yet some confabulations can have social affordances if carefully bounded. We propose critical confabulation (inspired by critical fabulation from literary and social theory), the use of LLM hallucinations to "fill-in-the-gap" for omissions in archives due to social and political inequality, and reconstruct divergent yet evidence-bound narratives for history's "hidden figures". We simulate these gaps with an open-ended narrative cloze task: asking LLMs to generate a masked event in a character-centric timeline sourced from a novel corpus of unpublished texts. We evaluate audited (for data contamination), fully-open models (the OLMo-2 family) and unaudited open-weight and proprietary baselines under a range of prompts designed to elicit controlled and useful hallucinations. Our findings validate LLMs' foundational narrative understanding capabilities to perform critical confabulation, and show how controlled and well-specified hallucinations can support LLM applications for knowledge production without collapsing speculation into a lack of historical accuracy and fidelity.

Critical Confabulation: Can LLMs Hallucinate for Social Good?

TL;DR

The paper addresses archival gaps caused by social inequality and proposes critical confabulation as a bounded narrative cloze framework that uses LLMs to reconstruct evidence-bound missing events in historical timelines. It grounds its evaluation in the BWTC dataset, creating ground-truth timelines for hidden figures and validating them through automatic perturbations and human annotation. Experimental results show that prompt design, especially event-type cues, can elicit meaningful confabulations, with GPT-5-chat achieving approaching 60% accuracy under strong prompting, while open-weight models also contribute substantially. The work highlights the potential of AI-assisted humanities research to surface latent histories while underscoring the need for robust evaluation, provenance tracking, and ethical safeguards as the approach scales across languages and archives.

Abstract

LLMs hallucinate, yet some confabulations can have social affordances if carefully bounded. We propose critical confabulation (inspired by critical fabulation from literary and social theory), the use of LLM hallucinations to "fill-in-the-gap" for omissions in archives due to social and political inequality, and reconstruct divergent yet evidence-bound narratives for history's "hidden figures". We simulate these gaps with an open-ended narrative cloze task: asking LLMs to generate a masked event in a character-centric timeline sourced from a novel corpus of unpublished texts. We evaluate audited (for data contamination), fully-open models (the OLMo-2 family) and unaudited open-weight and proprietary baselines under a range of prompts designed to elicit controlled and useful hallucinations. Our findings validate LLMs' foundational narrative understanding capabilities to perform critical confabulation, and show how controlled and well-specified hallucinations can support LLM applications for knowledge production without collapsing speculation into a lack of historical accuracy and fidelity.

Paper Structure

This paper contains 32 sections, 3 equations, 4 figures, 10 tables, 2 algorithms.

Table of Contents

  1. Background
  2. Task Setup
  3. Dataset
  4. Dataset Statistics
  5. Have LLMs Seen Our Unpublished Data?
  6. Ground Truth Construction
  7. Who Are the Hidden Figures of Our Dataset?
  8. Ground Truth Storyline Extraction
  9. Ground Truth Validation
  10. Experiments
  11. Narrative Cloze
  12. Evaluation
  13. Results
  14. Discussion
  15. Conclusion
  16. ...and 17 more sections

Figures (4)

  • Figure 1: Critical confabulation as an open-ended narrative-cloze task for LLMs.
  • Figure 2: Behavioral probe for contamination. Visualizing the cosine similarity decay (a) position-wise mean, (b) aggregate ECDF, (c) distribution across positions.
  • Figure 3: Ground-truth validation probes. (a) Partial-event cloze; (b) $n$-gram cloze.
  • Figure 4: Overview of the full critical confabulation workflow for reparative storytelling.