Capacity Constraints Make Admissions Processes Less Predictable

TL;DR

The paper tackles the challenge of predicting admissions outcomes under capacity constraints, where an individual’s likelihood of acceptance depends on the full applicant pool. It introduces instability and variability as formal measures of how admissions decisions change with small or large pool shifts, and connects these to the theory of choice functions, including $q$-acceptant, substitutability, and total-order queues. The authors develop a theoretical framework showing that standard ML representations can only capture highly idealized, pool-independent or single-queue processes, and they demonstrate that more complex, real-world programs exhibit greater instability and variability, making predictions under pool shifts unreliable. They validate the framework empirically using NYC high school admissions data simulated via a policy-based simulator, showing pronounced performance degradation as the applicant pool shifts, especially for programs with more queues; the work underscores the need for simulation-based or uncertainty-aware approaches when predicting admissions outcomes in capacity-constrained settings.

Abstract

Machine learning models are often used to make predictions about admissions process outcomes, such as for colleges or jobs. However, such decision processes differ substantially from the conventional machine learning paradigm. Because admissions decisions are capacity-constrained, whether a student is admitted depends on the other applicants who apply. We show how this dependence affects predictive performance even in otherwise ideal settings. Theoretically, we introduce two concepts that characterize the relationship between admission function properties, machine learning representation, and generalization to applicant pool distribution shifts: instability, which measures how many existing decisions can change when a single new applicant is introduced; and variability, which measures the number of unique students whose decisions can change. Empirically, we illustrate our theory on individual-level admissions data from the New York City high school matching system, showing that machine learning performance degrades as the applicant pool increasingly differs from the training data. Furthermore, there are larger performance drops for schools using decision rules that are more unstable and variable. Our work raises questions about the reliability of predicting individual admissions probabilities.

Figures (11)

• Figure 1: Model accuracy as a function of choice function instability, under increasing levels of distribution shift.
• Figure 2: Model accuracy as a function of choice function variability, under increasing levels of distribution shift.
• Figure 3: A set diagram with subsets labeled, for the proof of \ref{['thm:triangle-ineq']}.
• Figure 4: Model accuracy for each admissions method, under increasing levels of distribution shift.
• Figure 5: Replication of empirical performance figures using independent discretization.

Theorems & Definitions (66)

• Definition 1: $q$-Acceptance
• Definition 2: Total order
• Definition 3: Choice Distance
• Definition 4: $d$-Instability
• Definition 5: Variability
• Proposition 1: ML Representation
• Definition 6: Substitutability
• Theorem 1
• Definition 7: Sequential Composition
• Theorem 2