Efficient Online Set-valued Classification with Bandit Feedback

TL;DR

This work extends conformal prediction to online multiclass classification under bandit feedback, where ground-truth labels are not observed unless the pulled arm is correct. The authors introduce Bandit Class-specific Conformal Prediction (BCCP), which uses an unbiased estimator $\Delta_{t,k}$ and a SGD-based scheme to train a model and update class-specific quantiles $\tau_k$ for conformal sets. They prove $O(T^{-1/2})$ convergence for both class-specific coverage and the check-loss regret, and address hyper-parameter tuning with an Bandit Conformal with Experts ensemble to adapt learning rates. Empirically, BCCP with multiple score functions and policies demonstrates reliable class-specific coverage and shrinking prediction sets on CIFAR10/100 and SVHN, validating its applicability to online decision-making with restricted feedback.

Abstract

Conformal prediction is a distribution-free method that wraps a given machine learning model and returns a set of plausible labels that contain the true label with a prescribed coverage rate. In practice, the empirical coverage achieved highly relies on fully observed label information from data both in the training phase for model fitting and the calibration phase for quantile estimation. This dependency poses a challenge in the context of online learning with bandit feedback, where a learner only has access to the correctness of actions (i.e., pulled an arm) but not the full information of the true label. In particular, when the pulled arm is incorrect, the learner only knows that the pulled one is not the true class label, but does not know which label is true. Additionally, bandit feedback further results in a smaller labeled dataset for calibration, limited to instances with correct actions, thereby affecting the accuracy of quantile estimation. To address these limitations, we propose Bandit Class-specific Conformal Prediction (BCCP), offering coverage guarantees on a class-specific granularity. Using an unbiased estimation of an estimand involving the true label, BCCP trains the model and makes set-valued inferences through stochastic gradient descent. Our approach overcomes the challenges of sparsely labeled data in each iteration and generalizes the reliability and applicability of conformal prediction to online decision-making environments.

Figures (11)

• Figure 1: Accumulative cross-entropy loss under softmax policy and uniform policy.
• Figure 2: Flowchart of the online learning with bandit feedback. Here $\bm \tau^{t-1}=(\tau^{t-1}_1, \cdots, \tau^{t-1}_{|\mathcal{Y}|})^\top$.
• Figure 3: Performances under \ref{['alg']} with softmax policy. The black dotted lines in the bottom panel denote the oracle performance of the model with access to the full labels.
• Figure 4: Performances under \ref{['alg']} with uniform policy. The black dotted lines in the bottom panel denote the oracle performance of the model with access to the full labels.
• Figure 5: Performances under \ref{['alg:expert']} with softmax policy.

Theorems & Definitions (11)

• Proposition 3.1
• Theorem 3.2
• Corollary 3.3
• Theorem 3.4
• Corollary 3.5
• Theorem 3.6
• proof : Proof of \ref{['thm:cvg']}
• proof : Proof of \ref{['thm:checkReg']}
• Lemma 4.1
• Lemma 4.2