Calibrate-Extrapolate: Rethinking Prevalence Estimation with Black Box Classifiers

TL;DR

The paper introduces the Calibrate-Extrapolate framework for prevalence estimation with black-box classifiers, reframing the problem as learning the joint distribution of classifier outputs and ground truth on a base dataset and then extrapolating to a target dataset under stability assumptions. The calibration phase uses a labeled calibration sample to estimate a calibration curve Calib(·) and, together with the base classifier density, yields the base joint distribution P_B(C(x),y); disproportionate sampling can improve calibration efficiency. The extrapolation phase transfers the base joint distribution to a target by assuming either a stable calibration curve or stable class-conditional densities, employing techniques such as the probabilistic estimator or mixture-model/median-sweep, respectively. Through simulations with intrinsic and extrinsic data-generating processes and a real-world application to weekly toxicity prevalence on Reddit, X, and YouTube using Perspective API, the framework clarifies how different stability assumptions shape prevalence estimates and their robustness, offering practical guidance and noting limitations for diagnosing which stability assumption is most appropriate. The work demonstrates that while calibration aids base estimates, the extrapolation phase—and its assumed stability properties—crucially determines validity across datasets, with stronger classifiers providing more robustness under assumption violations.

Abstract

In computational social science, researchers often use a pre-trained, black box classifier to estimate the frequency of each class in unlabeled datasets. A variety of prevalence estimation techniques have been developed in the literature, each yielding an unbiased estimate if certain stability assumption holds. This work introduces a framework to rethink the prevalence estimation process as calibrating the classifier outputs against ground truth labels to obtain the joint distribution of a base dataset and then extrapolating to the joint distribution of a target dataset. We call this framework "Calibrate-Extrapolate". It clarifies what stability assumptions must hold for a prevalence estimation technique to yield accurate estimates. In the calibration phase, the techniques assume only a stable calibration curve between a calibration dataset and the full base dataset. This allows for the classifier outputs to be used for disproportionate random sampling, thus improving the efficiency of calibration. In the extrapolation phase, some techniques assume a stable calibration curve while some assume stable class-conditional densities. We discuss the stability assumptions from a causal perspective. By specifying base and target joint distributions, we can generate simulated datasets, as a way to build intuitions about the impacts of assumption violations. This also leads to a better understanding of how the classifier's predictive power affects the accuracy of prevalence estimates: the greater the predictive power, the lower the sensitivity to violations of stability assumptions in the extrapolation phase. We illustrate the framework with an application that estimates the prevalence of toxic comments on news topics over time on Reddit, Twitter/X, and YouTube, using Jigsaw's Perspective API as a black box classifier. Finally, we summarize several practical advice for prevalence estimation.

Figures (3)

• Figure 1: The Calibrate-Extrapolate framework. (top panel) The calibration phase aims to estimate the label prevalence for a base dataset $B$. We apply a black box classifier $C(\cdot)$ to the feature vector ${\boldsymbol x}\xspace$ of each item in $B$ and obtain a base classifier score density (step A $\rightarrow$ B). We curate a calibration sample $S$ from $B$ (step A $\rightarrow$ C). We then obtain the ground truth labels for $S$ (step C $\rightarrow$ D). Next, we fit a calibration curve function on $S$ (step D $\rightarrow$ E). We can use the calibration curve of $S$ and the observed classifier score density of $B$ to compute the joint distribution between $C({\boldsymbol x}\xspace)$ and $y$ for $B$ (step B + E $\rightarrow$ F). This is assuming calibration curves between $B$ and $S$ are stable, i.e., $P_S(y|C({\boldsymbol x}\xspace)) = P_B(y|C({\boldsymbol x}\xspace))$. From the joint distribution, we can derive the class-conditional densities for positive and negative class in $B$ (step H) and make a prevalence estimate of $B$ (step G). (bottom panel) The extrapolation phase aims to estimate the label prevalence in a target dataset $T$. We apply the same classifier $C(\cdot)$ to all items in $T$ (step I $\rightarrow$ J). By making a stability assumption (step K, N) that links the joint distributions of $B$ and $T$, we can infer the joint distribution of $T$ (step L, O, Q) and read off the final prevalence estimate of $T$ (step M, P, R).
• Figure 2: Calibration curves for two versions of Perspective API, both against the calibration sample from August 2021.
• Figure 3: Toxicity prevalence estimates using three alternative estimation techniques. Colored circled: assumed stable calibration curve; colored dashed: assumed stable class-conditional densities; black dashed: without calibration.