All Thresholds Barred: Direct Estimation of Call Density in Bioacoustic Data

TL;DR

This work addresses biased call counts from threshold-based detections in passive acoustic monitoring by proposing a threshold-free framework to directly estimate call density $P(\oplus)$. It introduces a log-bin validation scheme and Bayesian beta-binomial modeling to infer $P(\oplus)$, $P(\oplus|b)$, $P(b|\oplus)$, and related quantities, along with bootstrap-based confidence intervals and ROC-AUC estimation from validated distributions. The approach is tested on synthetic data, the Powdermill fully annotated dataset, and Hawaiian PAM data, with a focus on distribution shifts across sites and covariates. By leveraging the full distribution of classifier scores and stratified validation, the method reduces dependence on perfect classifiers and yields actionable site- and study-level density estimates for occupancy and ecological inference, offering practical implications for PAM management and future extensions to covariate-conditioned densities and call-type classifiers.

Abstract

Passive acoustic monitoring (PAM) studies generate thousands of hours of audio, which may be used to monitor specific animal populations, conduct broad biodiversity surveys, detect threats such as poachers, and more. Machine learning classifiers for species identification are increasingly being used to process the vast amount of audio generated by bioacoustic surveys, expediting analysis and increasing the utility of PAM as a management tool. In common practice, a threshold is applied to classifier output scores, and scores above the threshold are aggregated into a detection count. The choice of threshold produces biased counts of vocalizations, which are subject to false positive/negative rates that may vary across subsets of the dataset. In this work, we advocate for directly estimating call density: The proportion of detection windows containing the target vocalization, regardless of classifier score. Our approach targets a desirable ecological estimator and provides a more rigorous grounding for identifying the core problems caused by distribution shifts -- when the defining characteristics of the data distribution change -- and designing strategies to mitigate them. We propose a validation scheme for estimating call density in a body of data and obtain, through Bayesian reasoning, probability distributions of confidence scores for both the positive and negative classes. We use these distributions to predict site-level densities, which may be subject to distribution shifts. We test our proposed methods on a real-world study of Hawaiian birds and provide simulation results leveraging existing fully annotated datasets, demonstrating robustness to variations in call density and classifier model quality.

Figures (6)

• Figure 1: Root mean squared error (RMSE) between detection rates at various thresholds and true call density $P(\oplus)$, using synthetic data and a model with 0.9 ROC-AUC. Notice that the optimal threshold depends on $P(\oplus)$, which may vary across sites. Dotted lines indicate RMSE for the proposed validation scheme with 4 bins and 50 observations per bin.
• Figure 2: A schematic of our direct call density estimation method at (A) the study-level using our validation scheme and (B) the site- or covariate-level using computational Strategy 1.
• Figure 3: For different uninformative Beta distribution priors, we run simulation studies to find how often the ground-truth prediction of $P(\oplus)$ is in the predicted 90% confidence interval. The value $c=0.1$ has better coverage at low call density to the Jeffrey's prior or the uniform prior, both on (A) synthetic data and in (B) Powdermill simulations. All experiments use 4 bins and 50 observations per bin.
• Figure 4: Root mean squared error (RMSE) of the predicted $P(\oplus)$ in synthetic data, demonstrating that error steadily decreases as model quality improves, and above 0.75 ROC-AUC, and that logarithmic binning gives lower error above 0.75 ROC-AUC. RSME's are means over 50 trials. Solid lines report results with logarithmic binning, and dotted lines report results with standard, evenly spaced bins.
• Figure 5: The two user parameters for validation are the number of bins ($n_{bins}$) and the number of validation examples per bin ($k_{obs}$). Here we demonstrate, using the synthetic data harness, variation in the precision of $P(\oplus)$ as we vary the (A) number of bins and (B) observations per bin: Adding more data (more bins, or more observations) generally leads to lower root mean squared error (RMSE). For this model, at 0.9 ROC-AUC, error saturates at 4-6 bins, but decreases steadily as more observations per bin are added.