Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Relation Extraction Across Entire Books to Reconstruct Community Networks: The AffilKG Datasets

Erica Cai, Sean McQuade, Kevin Young, Brendan O'Connor

TL;DR

The paper tackles the challenge of evaluating how reliably relation extraction and KG construction from historical text support downstream graph analyses. It introduces AffilKG, a first-of-its-kind collection pairing complete scanned books with labeled affiliation graphs, and, for three datasets, expanded KGs with additional relation types, enabling both micro-level edge evaluation and macro-level graph analyses. Ground truth triplets are built via semi-automatic rule-based extraction with manual validation on archival social registers, and OCR is used to convert scans into text for extraction. Through a benchmarking pipeline that combines OCR with one-shot LLM prompts and evaluates multiple models, the study reveals substantial variability in edge-level accuracy and club-size estimates across datasets and models, highlighting the need to study error propagation in graph analyses. Overall, AffilKG provides a valuable resource for validating KG extraction methods in real-world social science research and guiding future improvements in OCR/RE pipelines.

Abstract

When knowledge graphs (KGs) are automatically extracted from text, are they accurate enough for downstream analysis? Unfortunately, current annotated datasets can not be used to evaluate this question, since their KGs are highly disconnected, too small, or overly complex. To address this gap, we introduce AffilKG (https://doi.org/10.5281/zenodo.15427977), which is a collection of six datasets that are the first to pair complete book scans with large, labeled knowledge graphs. Each dataset features affiliation graphs, which are simple KGs that capture Member relationships between Person and Organization entities -- useful in studies of migration, community interactions, and other social phenomena. In addition, three datasets include expanded KGs with a wider variety of relation types. Our preliminary experiments demonstrate significant variability in model performance across datasets, underscoring AffilKG's ability to enable two critical advances: (1) benchmarking how extraction errors propagate to graph-level analyses (e.g., community structure), and (2) validating KG extraction methods for real-world social science research.

Relation Extraction Across Entire Books to Reconstruct Community Networks: The AffilKG Datasets

TL;DR

The paper tackles the challenge of evaluating how reliably relation extraction and KG construction from historical text support downstream graph analyses. It introduces AffilKG, a first-of-its-kind collection pairing complete scanned books with labeled affiliation graphs, and, for three datasets, expanded KGs with additional relation types, enabling both micro-level edge evaluation and macro-level graph analyses. Ground truth triplets are built via semi-automatic rule-based extraction with manual validation on archival social registers, and OCR is used to convert scans into text for extraction. Through a benchmarking pipeline that combines OCR with one-shot LLM prompts and evaluates multiple models, the study reveals substantial variability in edge-level accuracy and club-size estimates across datasets and models, highlighting the need to study error propagation in graph analyses. Overall, AffilKG provides a valuable resource for validating KG extraction methods in real-world social science research and guiding future improvements in OCR/RE pipelines.

Abstract

When knowledge graphs (KGs) are automatically extracted from text, are they accurate enough for downstream analysis? Unfortunately, current annotated datasets can not be used to evaluate this question, since their KGs are highly disconnected, too small, or overly complex. To address this gap, we introduce AffilKG (https://doi.org/10.5281/zenodo.15427977), which is a collection of six datasets that are the first to pair complete book scans with large, labeled knowledge graphs. Each dataset features affiliation graphs, which are simple KGs that capture Member relationships between Person and Organization entities -- useful in studies of migration, community interactions, and other social phenomena. In addition, three datasets include expanded KGs with a wider variety of relation types. Our preliminary experiments demonstrate significant variability in model performance across datasets, underscoring AffilKG's ability to enable two critical advances: (1) benchmarking how extraction errors propagate to graph-level analyses (e.g., community structure), and (2) validating KG extraction methods for real-world social science research.
Paper Structure (11 sections, 1 equation, 1 figure, 7 tables)

This paper contains 11 sections, 1 equation, 1 figure, 7 tables.

Table of Contents

  1. Introduction
  2. AffilKG Dataset Construction
  3. Benchmarking LLMs for Extracting Local and Global Structure
  4. Conclusion
  5. Appendix
  6. Dataset licensing
  7. Text and knowledge graph information
  8. Top 10 largest organizations
  9. Details for constructing regular expressions to extract ground truth triplets
  10. LLM prompts
  11. Details for evaluating triplets

Figures (1)

  • Figure 1: Pipeline of converting scans of historical text to affiliation graphs, where highlighted text indicates clubs.