Prediction-powered estimators for finite population statistics in highly imbalanced textual data: Public hate crime estimation
Hannes Waldetoft, Jakob Torgander, Måns Magnusson
TL;DR
The paper tackles estimating finite-population statistics from large text corpora when labels are costly to obtain. It proposes a prediction-powered framework that uses transformer-based classifier outputs as auxiliary information in standard survey estimators, notably the Hansen-Hurwitz, stratified, and difference estimators. The authors formalize the H2P2 estimator, discuss stratification-by-prediction, and address variance behavior as classifier quality improves, then apply the method to Swedish hate crime statistics derived from police reports. Empirical results show substantial efficiency gains (design effects around 0.0068) and provide unbiased estimates of total hate crimes and police under-reporting, illustrating the approach’s practical value for official statistics and similar applications.
Abstract
Estimating population parameters in finite populations of text documents can be challenging when obtaining the labels for the target variable requires manual annotation. To address this problem, we combine predictions from a transformer encoder neural network with well-established survey sampling estimators using the model predictions as an auxiliary variable. The applicability is demonstrated in Swedish hate crime statistics based on Swedish police reports. Estimates of the yearly number of hate crimes and the police's under-reporting are derived using the Hansen-Hurwitz estimator, difference estimation, and stratified random sampling estimation. We conclude that if labeled training data is available, the proposed method can provide very efficient estimates with reduced time spent on manual annotation.