Optimal Policy Learning with Observational Data in Multi-Action Scenarios: Estimation, Risk Preference, and Potential Failures

TL;DR

OPL from observational data is framed in a contextual multi-armed bandit with $N$ observations, $J+1$ actions, outcomes $Y_i(j)$ and conditional means $\mu_i(j,\mathbf{x}_i) = E(Y_i(j)\mid \mathbf{x}_i)$, with value $V(\pi)=E[Y(\pi(\mathbf{X}))]$ and first-best policy $\pi^*$; estimation relies on regression adjustment, inverse propensity weighting, and doubly robust estimators under unconfoundedness $A1$ and overlap $A2$. The paper extends offline and online learning to incorporate risk by modeling reward uncertainty via $\sigma^2(j,\mathbf{x})=\mathrm{Var}(Y\mid D=j,\mathbf{x})$ and risk-averse utilities, showing that risk attitudes can reorder actions and affect regret. It also analyzes identification vulnerabilities from weak overlap or weak unconfoundedness and provides empirical illustrations on a job-training application, illustrating when data-driven policies may misallocate resources. Overall, the work integrates estimation, risk-aware decision-making, and robustness considerations into OPL with observational data, offering guidance on method choice, risk modeling, and diagnostic checks for practical deployment.

Abstract

This paper deals with optimal policy learning (OPL) with observational data, i.e. data-driven optimal decision-making, in multi-action (or multi-arm) settings, where a finite set of decision options is available. It is organized in three parts, where I discuss respectively: estimation, risk preference, and potential failures. The first part provides a brief review of the key approaches to estimating the reward (or value) function and optimal policy within this context of analysis. Here, I delineate the identification assumptions and statistical properties related to offline optimal policy learning estimators. In the second part, I delve into the analysis of decision risk. This analysis reveals that the optimal choice can be influenced by the decision maker's attitude towards risks, specifically in terms of the trade-off between reward conditional mean and conditional variance. Here, I present an application of the proposed model to real data, illustrating that the average regret of a policy with multi-valued treatment is contingent on the decision-maker's attitude towards risk. The third part of the paper discusses the limitations of optimal data-driven decision-making by highlighting conditions under which decision-making can falter. This aspect is linked to the failure of the two fundamental assumptions essential for identifying the optimal choice: (i) overlapping, and (ii) unconfoundedness. Some conclusions end the paper.

Figures (16)

• Figure 1: Threshold-based policy class.
• Figure 2: A heuristic representation of the model's architecture.
• Figure 3: Computation of the optimal choice when a new environment signal comes up according to Procedure 2, under assumptions A1 and A2.
• Figure 4: Reward distribution and uncertainty realtive to two action, A and B. Action A provides a lower average return, but with smaller uncertainty. Action B provides a higher average return, but with larger uncertainty.
• Figure 5: Example of actions' preferential ordering. Under a linear risk-averse preferences, the agent prefers action A over action B. Under a quadratic risk-averse preferences, the agent is indifferent between action A and B.