EnsembleLink: Accurate Record Linkage Without Training Data

TL;DR

EnsembleLink tackles the uncertainty inherent in record linkage by introducing a zero-shot retrieve-and-rerank pipeline that leverages pre-trained semantic models to learn cross-domain relationships without task-specific labels. It fuses dense embeddings and sparse lexical signals for high-recall candidate retrieval and applies a cross-encoder to produce a match score $s(q,c) \in [0,1]$, with an optional LLM reranker for world-knowledge cases; all components run locally. Across tasks spanning city names, person names, organizations, multilingual parties, and DBLP-Scholar, EnsembleLink matches or surpasses supervised methods while avoiding labeled data and external API calls. The approach delivers robust accuracy, strong scalability on consumer hardware, and practical applicability for researchers needing reproducible, privacy-preserving linkage. These contributions offer a principled, label-free alternative to traditional probabilistic and supervised record-linkage methods with broad utility for empirical social science and related fields.

Abstract

Record linkage, the process of matching records that refer to the same entity across datasets, is essential to empirical social science but remains methodologically underdeveloped. Researchers treat it as a preprocessing step, applying ad hoc rules without quantifying the uncertainty that linkage errors introduce into downstream analyses. Existing methods either achieve low accuracy or require substantial labeled training data. I present EnsembleLink, a method that achieves high accuracy without any training labels. EnsembleLink leverages pre-trained language models that have learned semantic relationships (e.g., that "South Ozone Park" is a neighborhood in "New York City" or that "Lutte ouvriere" refers to the Trotskyist "Workers' Struggle" party) from large text corpora. On benchmarks spanning city names, person names, organizations, multilingual political parties, and bibliographic records, EnsembleLink matches or exceeds methods requiring extensive labeling. The method runs locally on open-source models, requiring no external API calls, and completes typical linkage tasks in minutes.