Driving down Poisson error can offset classification error in clinical tasks

TL;DR

The paper addresses how Poisson variability from small clinical sample volumes can limit the effective performance of clinicians and ML systems alike. It argues that automated systems can offset imperfect object-level accuracy by increasing the examined volume $V$, thereby reducing Poisson error and achieving total error comparable to a perfectly accurate human constrained by protocol volumes. A mathematical toolkit is developed to relate Poisson error, classification error, and total error, including key relations such as $P(n \geq 1 \mid V, N) \geq 0.95$ for limit of detection and a detailed quantification error formula (Eq. (eqnStdQuantError)) that incorporates $V_{SE}$, $\mu(S)$, $\sigma(S)$, $\mu(F)$, and $\sigma(F)$. Demonstrations on malaria diagnosis and quantitation show that, under reasonable parameter choices, enlarging the sampled volume (e.g., to ~0.2–0.4 $\mu L$) can align ML total error with the standard of care across most parasitemia ranges. The framework provides actionable design guidance for deploying ML in low-resource clinical settings, highlighting throughput as a distinct axis of improvement with broad applicability beyond malaria to any domain with Poisson variability.

Abstract

Medical machine learning algorithms are typically evaluated based on accuracy vs. a clinician-defined ground truth, a reasonable initial choice since trained clinicians are usually better classifiers than ML models. However, this metric does not fully capture the actual clinical task: it neglects the fact that humans, even with perfect accuracy, are subject to non-trivial error from the Poisson statistics of rare events, because clinical protocols often specify a relatively small sample size. For example, to quantitate malaria on a thin blood film a clinician examines only 2000 red blood cells (0.0004 uL), which can yield large Poisson variation in the actual number of parasites present, so that a perfect human's count can differ substantially from the true average load. In contrast, an ML system may be less accurate on an object level, but it may also have the option to examine more blood (e.g. 0.1 uL, or 250x). Then while its parasite identification error is higher, the Poisson variability of its estimate is lower due to larger sample size. To qualify for clinical deployment, an ML system's performance must match current standard of care, typically a very demanding target. To achieve this, it may be possible to offset the ML system's lower accuracy by increasing its sample size to reduce Poisson error, and thus attain the same net clinical performance as a perfectly accurate human limited by smaller sample size. In this paper, we analyse the mathematics of the relationship between Poisson error, classification error, and total error. This mathematical toolkit enables teams optimizing ML systems to leverage a relative strength (larger sample sizes) to offset a relative weakness (classification accuracy). We illustrate the methods with two concrete examples: diagnosis and quantitation of malaria on blood films.

Figures (4)

• Figure 1: Poisson distributions showing number of parasites actually found in 500 WBCs' worth of blood, given various true parasitemias. The LoD is $\approx$50 p/$\mu L$.
• Figure 2: Estimating the volume required to match an LoD = 70, given $\mu (\textit{S})$ = 0.85 and $\sigma (\textit{F}) = 10 / \mu L$. The required number of TPs for each volume are shown as red dots. For $V = 0.2~ \mu L$, 95% of cases will exceed the required number.
• Figure 3: The size of various terms in the standard error of quantitation, Equation \ref{['eqnStdQuantError']}. At low $p$ errors due to Poisson and $\sigma (\textit{F})$ dominate, while at high $p$ errors due to $\sigma (\textit{S})$ and $V_{SE}$ dominate. See Python code in Appendix \ref{['appendixStdErrorQuantCode']}.
• Figure 4: Standard error of quantitation vs. parasitemia, for: (In RED) Humans with perfect accuracy, one without and one with volume estimation error, examining protocol volumes; (In BLUE) An imperfect algorithm with performance per the text, at various examined volumes $V$. Larger $V$ compensates for its classification and volume errors.