Diversifying Conformal Selections

TL;DR

The key idea of DACS is to use optimal stopping theory to adaptively choose the set of e-values which (approximately) maximizes the expected diversity measure, and develops a number of computational heuristics which greatly improve its running time for generic diversity metrics.

Abstract

When selecting from a list of potential candidates, it is important to ensure not only that the selected items are of high quality, but also that they are sufficiently dissimilar so as to both avoid redundancy and to capture a broader range of desirable properties. In drug discovery, scientists aim to select potent drugs from a library of unsynthesized candidates, but recognize that it is wasteful to repeatedly synthesize highly similar compounds. In job hiring, recruiters may wish to hire candidates who will perform well on the job, while also considering factors such as socioeconomic background, prior work experience, gender, or race. We study the problem of using any prediction model to construct a maximally diverse selection set of candidates while controlling the false discovery rate (FDR) in a model-free fashion. Our method, diversity-aware conformal selection (DACS), achieves this by designing a general optimization procedure to construct a diverse selection set subject to a simple constraint involving conformal e-values which depend on carefully chosen stopping times. The key idea of DACS is to use optimal stopping theory to adaptively choose the set of e-values which (approximately) maximizes the expected diversity measure. We give an example diversity metric for which our procedure can be run exactly and efficiently. We also develop a number of computational heuristics which greatly improve its running time for generic diversity metrics. We demonstrate the empirical performance of our method both in simulation and on job hiring and drug discovery datasets.

Figures (44)

• Figure 1: Tradeoff between diversity and nominal FDR level $\alpha$ on a job hiring dataset discussed in Section \ref{['expers:hiring']}. As $\alpha$ increases, the achieved diversity by our method (blue) increases and, in particular, dominates that of the conformal selection (CS) jin2023selection method (red) at the same nominal level.
• Figure 2: Illustration of the variables involved in defining the filtration as well as variables defining filtration at time $2$, shown for an example with $n=2$ calibration points (blue) and $m=2$ test points (red).
• Figure 3: High-level overview of DACS procedure. Flowchart illustrates the five main steps of our method, with panels (a)--(e) detailing each step. The most computationally intensive parts are the reward computation (panel c) and e-value optimization (panel e). As shown in the brown panel, both steps can be performed exactly and efficiently for the underrepresentation index; for general diversity metrics, we introduce heuristics to accelerate computation.
• Figure 4: Example paths for warm starting in a problem with $n=5$ calibration points and $m=5$ test points. Upper panel illustrates which scores, after sorting, are known to belong to the calibration set (blue), the test set (red), or are unknown (grey) at the BH stopping time $\tau_{\textnormal{BH}}$. The lower panel visualizes the table of $R_t(s_t)$ values for each $s_t \in \Omega_t, t \in [\tau_{\textnormal{BH}}]$. White squares indicate $(s_t,t)$ pairs for which $s_t \not\in \Omega_t$ and hence the value need not be computed. Otherwise, our warm starting heuristic fills out cells by following the orange arrows along the purple gradients.
• Figure 5: Left: Conditional average cluster proportions given selection set is non-empty for DACS selection set (blue) compared to CS selection set (red) for job hiring dataset at various nominal levels $\alpha$ and different settings (i.e., values of $n$ and $m$); horizontal lines, when visible, denote the average proportion of selection sets that are empty. Right: average number of rejections made by each method for various levels $\alpha$ and different settings of $n$ and $m$.

Theorems & Definitions (35)

• Example 1.1: Sharpe ratio
• Example 1.2: Markowitz objective
• Example 1.3: Underrepresentation index
• Corollary 2.1
• Remark 1
• Proposition 2.1
• Proposition 2.2
• Proposition 2.3
• Theorem 2.3
• Proposition 3.1