Outlier Ranking in Large-Scale Public Health Streams

TL;DR

This work tackles the problem of ranking the most important outliers across thousands of public health data streams by introducing Outshines, a hierarchical-network plus extreme value analysis method that calibrates per-stream univariate outlier outputs. By constructing a daily empirical reference distribution $ P_{i,t}$ through block maxima across geospatial siblings within a regime, Outshines delivers high-resolution, comparable rankings $y(d)$ that reflect recent historical behavior. In expert evaluations and real deployments, Outshines consistently outperformed threshold and sibling ranking approaches on standard metrics (e.g., AUC, rank correlation) and reduced maximally tied outliers, enabling faster, more reliable identification of events worth investigating—reported as up to $9.1\times$ faster than baselines. The approach is model-agnostic with respect to the chosen univariate detector and is deployed in public health settings, offering scalable applicability to other domains with large-scale, nonstationary data streams.

Abstract

Disease control experts inspect public health data streams daily for outliers worth investigating, like those corresponding to data quality issues or disease outbreaks. However, they can only examine a few of the thousands of maximally-tied outliers returned by univariate outlier detection methods applied to large-scale public health data streams. To help experts distinguish the most important outliers from these thousands of tied outliers, we propose a new task for algorithms to rank the outputs of any univariate method applied to each of many streams. Our novel algorithm for this task, which leverages hierarchical networks and extreme value analysis, performed the best across traditional outlier detection metrics in a human-expert evaluation using public health data streams. Most importantly, experts have used our open-source Python implementation since April 2023 and report identifying outliers worth investigating 9.1x faster than their prior baseline. Other organizations can readily adapt this implementation to create rankings from the outputs of their tailored univariate methods across large-scale streams.

Figures (5)

• Figure 1: Existing methods identify 14-20k tied maximum-priority outliers (a subset shown here) from 3-4m points. Outshines ranking enables experts to prioritize outliers.
• Figure 2: The geospatial hierarchy for public health streams covers 4270 regions. HRRs may serve multiple states like HRR 112 and 225 serve both D.E. and M.D. residents.
• Figure 3: Outshines models the hierarchical relationships between streams, applies a univariate method per stream to calculate $\phi$s, creates $\mathcal{P}_{i, t}$ via block maxima per indicator $i$ per day $t$ across sibling streams, and finally, ranks $\phi_t$ by its quantile in $\mathcal{P}_{i,t}$.
• Figure 4: Evaluation Metrics: The red box highlights the best performing combination ranking method x tailored univariate method for standard binary and ranking metrics. Correlations were N/A when the method returned all 0s or 1s.
• Figure 5: Experts can identify outliers worth investigating more quickly with Outshines.