ALER: An Active Learning Hybrid System for Efficient Entity Resolution

TL;DR

ALER tackles the label scarcity and scalability challenges in entity resolution by employing a frozen bi-encoder to produce static semantic embeddings, a K-Means-based semantic partitioning scheme, and a lightweight Siamese MLP within a partitioned active-learning loop. It introduces a hybrid query strategy that blends informative (uncertain) and confident (error-correcting) samples and a two-stage cascade for fast recall followed by precise lexical refinement. The approach achieves state-of-the-art F1 scores while dramatically reducing training and resolution times, and scales effectively to multi-million-record datasets. This combination provides a practical, label-efficient ER solution suitable for real-world data integration workloads.

Abstract

Entity Resolution (ER) is a critical task for data integration, yet state-of-the-art supervised deep learning models remain impractical for many real-world applications due to their need for massive, expensive-to-obtain labeled datasets. While Active Learning (AL) offers a potential solution to this "label scarcity" problem, existing approaches introduce severe scalability bottlenecks. Specifically, they achieve high accuracy but incur prohibitive computational costs by re-training complex models from scratch or solving NP-hard selection problems in every iteration. In this paper, we propose ALER, a novel, semi-supervised pipeline designed to bridge the gap between semantic accuracy and computational scalability. ALER eliminates the training bottleneck by using a frozen bi-encoder architecture to generate static embeddings once and then iteratively training a lightweight classifier on top. To address the memory bottleneck associated with large-scale candidate pools, we first select a representative sample of the data and then use K-Means to partition this sample into semantically coherent chunks, enabling an efficient AL loop. We further propose a hybrid query strategy that combines "confused" and "confident" pairs to efficiently refine the decision boundary while correcting high-confidence errors.Extensive evaluation demonstrates ALER's superior efficiency, particularly on the large-scale DBLP dataset: it accelerates the training loop by 1.3x while drastically reducing resolution latency by a factor of 3.8 compared to the fastest baseline.

Figures (4)

• Figure 1: The ALER pipeline
• Figure 2: Effect of Chunk Count $(N)$ on F1-Score $(B=300)$
• Figure 3: Effect of Budget Size $(B)$ on Final F1-Score (max $8$ Iterations)
• Figure 4: Query Strategy Effectiveness $(B=300)$