The Initial Screening Order Problem

TL;DR

The paper addresses how the initial screening order (ISO) influences the selection of a size-$k$ candidate set under two search formulations, best-$k$ and good-$k$, and with a representational quota $q$. It formalizes a utility-based framework, introduces an algorithmic screener and a human-like screener with fatigue and position bias, and provides two sequential search procedures (ExaminationSearch and CascadeSearch) to study ISO effects via Monte Carlo simulations. The key finding is that ISO can degrade individual fairness for human-like screeners while preserving group fairness, and that partial-search dynamics under ISO can hinder optimality depending on score distributions and correlations between ISO and scores. The work also delivers a flexible simulation framework that can model multiple screening tasks and guide practitioners in mitigating ISO-induced biases before conducting real-world screening. Overall, the study highlights the importance of carefully designing the ISO and screening procedures to balance fairness and efficiency in candidate selection.

Abstract

We investigate the role of the initial screening order (ISO) in candidate screening. The ISO refers to the order in which the screener searches the candidate pool when selecting $k$ candidates. Today, it is common for the ISO to be the product of an information access system, such as an online platform or a database query. The ISO has been largely overlooked in the literature, despite its impact on the optimality and fairness of the selected $k$ candidates, especially under a human screener. We define two problem formulations describing the search behavior of the screener given an ISO: the best-$k$, where it selects the top $k$ candidates; and the good-$k$, where it selects the first good-enough $k$ candidates. To study the impact of the ISO, we introduce a human-like screener and compare it to its algorithmic counterpart, where the human-like screener is conceived to be inconsistent over time. Our analysis, in particular, shows that the ISO, under a human-like screener solving for the good-$k$ problem, hinders individual fairness despite meeting group fairness, and hampers the optimality of the selected $k$ candidates. This is due to position bias, where a candidate's evaluation is affected by its position within the ISO. We report extensive simulated experiments exploring the parameters of the best-$k$ and good-$k$ problems for both screeners. Our simulation framework is flexible enough to account for multiple candidate screening tasks, being an alternative to running real-world procedures.

Figures (5)

• Figure 1: ExaminationSearch
• Figure 2: Upper. Left: densities of candidate score distributions, with scores raging from 0 to 1. Center and Right: resp., RtB and JdS for all score distributions over $\psi$ under $\theta \mathrel{{\hbox{$\perp\mkern-10mu\perp$}}} s$. Lower. Left: $n, k$ combinations for $tN(0.5, 0.02)$ distribution over $\Psi$ under $\theta \mathrel{{\hbox{$\perp\mkern-10mu\perp$}}} s$. Center and Right: resp., RtB under different $\rho$'s (i.e., under $\theta \not\!\perp\!\!\!\perp s$) for $tN(0.5, 0.02)$ and $tN(1, 0.05)$ distributions over $\Psi$.
• Figure 3: Upper. RtB for different candidate score distributions under $\theta \mathrel{{\hbox{$\perp\mkern-10mu\perp$}}} s$. Left: good-$k$ solution for fatigue with $\epsilon_1$ over $\psi$. Center: good-$k$ solution for fatigue with $\epsilon_2$ over $\psi$. Right: best-$k$ solution for fatigue with $\epsilon_1$ over $q$. Lower. RtB for different $\rho$'s (i.e., under $\theta \not\!\perp\!\!\!\perp s$) over $\psi$. Left: good-$k$ solution for fatigue with $\epsilon_1$ and $tN(0.5, 0.02)$ distribution. Center: good-$k$ solution for fatigue with $\epsilon_2$ and $tN(0.5, 0.02)$ distribution. Right: good-$k$ solution for fatigue with $\epsilon_1$ and $tN(1, 0.05)$ distribution.
• Figure 4: HumanLikeExaminationSearch
• Figure 5: Left: fraction of protected candidates in the solution of good-$k$ for different representational quotas $q$, for the $tN(0.5, 0.02)$ score distribution and with setting $n=400$, $k=20$, $\theta \mathrel{{\hbox{$\perp\mkern-10mu\perp$}}} s$. Center: RtB for different representational quotas $q$, for the $tN(0.5, 0.02)$ score distribution and with setting $n=400$, $k=20$, $\theta \mathrel{{\hbox{$\perp\mkern-10mu\perp$}}} s$. Right: RtB for different representational quotas $q$, for the $tN(1, 0.05)$ score distribution and with setting $n=400$, $k=20$, $\theta \mathrel{{\hbox{$\perp\mkern-10mu\perp$}}} s$.

Theorems & Definitions (2)

• Remark 1
• Remark 2