Cultural Analytics for Good: Building Inclusive Evaluation Frameworks for Historical IR

TL;DR

The paper addresses equitable access to historical knowledge by developing an inclusive evaluation framework for historical IR grounded in cultural analytics. It builds a large-scale benchmark on the British Library BL19 collection (> $35{,}000$ works from $1700$–$1899$) by combining expert-designed queries, LLM-assisted relevance judgments, and expert validation. It demonstrates cross-genre knowledge transfer from 19th‑century fiction to non-fiction, showing that fiction-informed relevance models can improve non-fiction retrieval while maintaining interpretability. The authors provide a reproducible benchmark, code, and analysis that support more transparent, culturally aware, and emancipatory digital archives. The work offers a methodological paradigm for retrieval systems that balance accuracy with equity and cultural context.

Abstract

This work bridges the fields of information retrieval and cultural analytics to support equitable access to historical knowledge. Using the British Library BL19 digital collection (more than 35,000 works from 1700-1899), we construct a benchmark for studying changes in language, terminology and retrieval in the 19th-century fiction and non-fiction. Our approach combines expert-driven query design, paragraph-level relevance annotation, and Large Language Model (LLM) assistance to create a scalable evaluation framework grounded in human expertise. We focus on knowledge transfer from fiction to non-fiction, investigating how narrative understanding and semantic richness in fiction can improve retrieval for scholarly and factual materials. This interdisciplinary framework not only improves retrieval accuracy but also fosters interpretability, transparency, and cultural inclusivity in digital archives. Our work provides both practical evaluation resources and a methodological paradigm for developing retrieval systems that support richer, historically aware engagement with digital archives, ultimately working towards more emancipatory knowledge infrastructures.

Figures (1)

• Figure 1: Comparisons of per query performance, measured by Average Precision (AP), among the three models from Table \ref{['tab:result']}: NonFiction$_{base}$, NonFiction$_{RLM}$ and Fiction$_{RLM}$.