Policy Learning with Competing Agents

TL;DR

This work tackles learning treatment assignment under capacity constraints when agents respond strategically, introducing a dynamic model where a fixed selection criterion β and a time-varying threshold S^{t} produce equilibrium behavior. Through mean-field analysis, it proves the existence and uniqueness of a function s(β) that maps the criterion to an equilibrium threshold, and provides conditions under which fixed-point iterations converge; it also develops a consistent estimator for the equilibrium policy value gradient using unit-level local experiments. The finite-sample extension shows stochastic equilibria in finite populations converge to the mean-field equilibrium as n grows, with concentration guarantees for the threshold. Empirically, the authors validate their gradient-based learning approach in a semi-synthetic NELS:88-based experiment, demonstrating that competition-aware optimization yields higher equilibrium welfare than capacity-aware or strategy-aware baselines, with policy distributions adapting to heterogeneous agent types. Overall, the paper advances policy learning in the presence of strategic competition and capacity limits, offering tractable estimation, learning algorithms, and practical insights for applications like college admissions and hiring.

Abstract

Decision makers often aim to learn a treatment assignment policy under a capacity constraint on the number of agents that they can treat. When agents can respond strategically to such policies, competition arises, complicating estimation of the optimal policy. In this paper, we study capacity-constrained treatment assignment in the presence of such interference. We consider a dynamic model where the decision maker allocates treatments at each time step and heterogeneous agents myopically best respond to the previous treatment assignment policy. When the number of agents is large but finite, we show that the threshold for receiving treatment under a given policy converges to the policy's mean-field equilibrium threshold. Based on this result, we develop a consistent estimator for the policy gradient. In a semi-synthetic experiment with data from the National Education Longitudinal Study of 1988, we demonstrate that this estimator can be used for learning capacity-constrained policies in the presence of strategic behavior.

Figures (7)

• Figure 1: We plot $\omega_{i}(s; \beta)$ vs. $s$ at different noise levels and observe its continuity and contraction properties.
• Figure 2: We plot iterates an example process \ref{['eq:emp_fpi']}. For large $n$, iterates concentrate about the mean-field equilibrium.
• Figure 3: We plot the equilibrium policy value obtained from iterates of strategy-aware and competition-aware methods.
• Figure 4: We plot the score distributions that are induced by $\beta_{\text{comp}}$, $\beta_{\text{strat}}$, $\beta_{\text{cap}}$ to maximize $V_{\text{eq}, j}(\beta)$ for $j=1, 2, 3$. We plot a histogram of scores for each agent with a distinct unobservable $(Z_{i}, c_{i})$. Agents are color-coded on according to low (yellow), medium (orange), and high (pink) relative SES. The selection criterion $\beta_{\text{comp}}$ accepts students with varying SES.
• Figure 5: We cluster the agent unobservables computed from the NELS dataset into $K=8$ clusters with $K$-means clustering. Using t-SNE, we visualize two-dimensional embeddings of unobservables $\{( Z_{i}, G_{i}, Y_{i, 1}(1), Y_{i, 2}(1)), Y_{i, 3}(1)\}_{i=1}^{n}$ from the NELS data. Each point represents an two-dimensional embedding of $(Z_{i}, G_{i}, Y_{i, 1}(1), Y_{i, 2}(1), Y_{i, 3}(1))$ and the color of the point corresponds to the cluster that the agent belongs to.

Theorems & Definitions (56)

• Definition 1: Equilibrium Policy Value
• Proposition 1
• Theorem 2
• Lemma 3
• Theorem 4
• Corollary 5
• Lemma 6
• Definition 2: Model Gradient
• Definition 3: Equilibrium Gradient
• Definition 4: Policy Gradient