The Good, the Bad, and the Sampled: a No-Regret Approach to Safe Online Classification

TL;DR

The paper tackles safe online binary classification under an unknown logistic model and unknown context distribution by proposing SCOUT, a no-regret algorithm that interleaves label collection with distribution estimation. SCOUT computes a data-driven, conservative testing threshold to decide when to test, and uses a sample-splitting scheme to learn both ${\theta^*}$ and ${P}$. The authors prove high-probability safety guarantees and sublinear excess testing ${O(\sqrt{T})}$ relative to an oracle, yielding a tilde-${O}(d\sqrt{T})$ regret bound after accounting for problem geometry. Simulations corroborate the theory, showing efficient parameter estimation while maintaining safety, making the method attractive for cost-sensitive medical screening and other high-stakes sequential decision tasks.

Abstract

We study the problem of sequentially testing individuals for a binary disease outcome whose true risk is governed by an unknown logistic model. At each round, a patient arrives with feature vector $x_t$, and the decision maker may either pay to administer a (noiseless) diagnostic test--revealing the true label--or skip testing and predict the patient's disease status based on their feature vector and prior history. Our goal is to minimize the total number of costly tests required while guaranteeing that the fraction of misclassifications does not exceed a prespecified error tolerance $α$, with probability at least $1-δ$. To address this, we develop a novel algorithm that interleaves label-collection and distribution estimation to estimate both $θ^{*}$ and the context distribution $P$, and computes a conservative, data-driven threshold $τ_t$ on the logistic score $|x_t^\topθ|$ to decide when testing is necessary. We prove that, with probability at least $1-δ$, our procedure does not exceed the target misclassification rate, and requires only $O(\sqrt{T})$ excess tests compared to the oracle baseline that knows both $θ^{*}$ and the patient feature distribution $P$. This establishes the first no-regret guarantees for error-constrained logistic testing, with direct applications to cost-sensitive medical screening. Simulations corroborate our theoretical guarantees, showing that in practice our procedure efficiently estimates $θ^{*}$ while retaining safety guarantees, and does not require too many excess tests.

Figures (5)

• Figure 1: Threshold-based testing policy.
• Figure 2: Pessimistic choice of $|\tau_t|$.
• Figure 3: Simulation results. First and second row correspond to $d=2$, where the first row shows $\alpha=0.05$, and the second $\alpha=0.1$. Third row shows $d=8,\alpha=0.1$. $x$-axis corresponds to time (round number). Left plots show the cumulative test rate (10-90% quantiles shaded), where blue shows the performance of SCOUT , with the oracle test rate shown in orange at $p^\star$. Empirical test rate for optimal threshold policy plotted in green. The middle plots show the excess number of tests, demonstrating the sublinear regret of SCOUT . The right plots show the misclassification rate of SCOUT . While the optimal baseline policy fluctuates around the desired threshold $\alpha$, often exceeding it, SCOUT starts far below (very safe) then gradually learns to be more aggressive, approaching misclassification rate $\alpha$ but never exceeding it.
• Figure 4: ${\mathcal{B}}(0,1) \cap \{x:|\langle x, \theta^\star \rangle|\le \tau^\star\}$
• Figure 5: The CDF of $Beta(\frac{d+1}{2},\frac{1}{2})$ for various values of $d$.

Theorems & Definitions (58)

• Definition 1
• Proposition 1
• Definition 2
• Lemma 1
• Lemma 2
• Definition 3
• Lemma 3
• Lemma 4
• Theorem 1
• proof