2-Step Agent: A Framework for the Interaction of a Decision Maker with AI Decision Support

TL;DR

A general computational framework, the 2-Step Agent, which models the effects of AI-assisted decision making and reveals several potential pitfalls of AI-driven decision support and highlights the need for thorough model documentation and proper user training.

Abstract

Across a growing number of fields, human decision making is supported by predictions from AI models. However, we still lack a deep understanding of the effects of adoption of these technologies. In this paper, we introduce a general computational framework, the 2-Step Agent, which models the effects of AI-assisted decision making. Our framework uses Bayesian methods for causal inference to model 1) how a prediction on a new observation affects the beliefs of a rational Bayesian agent, and 2) how this change in beliefs affects the downstream decision and subsequent outcome. Using this framework, we show by simulations how a single misaligned prior belief can be sufficient for decision support to result in worse downstream outcomes compared to no decision support. Our results reveal several potential pitfalls of AI-driven decision support and highlight the need for thorough model documentation and proper user training.

Figures (6)

• Figure 1: The diagram shows the overall flow of the model. A new patient with certain attributes (1) arrives for treatment, and the patient's data (2), potentially along with the output of a prediction model (2'), is observed by the agent, e.g., a doctor. The agent uses these observations to update their previous beliefs about the population and the predictive model (3), a step which we model as a Bayesian update. Based on their new beliefs, the agent estimates the effect of treatment (4) and makes a treatment decision (5). Finally, the treatment together with the patient's attributes determine a final outcome (6).
• Figure 2: The models involved in the paper. Model (b) represents the SCM generating the data in the real world, while models (a) and (c) are the first and second steps of the 2-Step Agent. Model (a) allows the agent to perform a belief update given new information from ML-DS, while model (c) allows the agent to estimate the effect of interventions on new data points.
• Figure 3: Figures displaying the effects of different agent priors when using ML-DS (in orange) and without it (in blue). In all plots except from $N_E$, the x-axis represents the deviation of the agent's prior belief from the historical SCM's parameters. The top row displays the agent's prior and posterior (predictive) belief of CATE, while the bottom row shows the impact of the agent's decisions on the downstream outcome $Y$. The left column pertains to the belief on treatment effect, $N_E$, the central column the belief about past treatment policy ($\mu_A$) and the right column about the belief on past covariate distribution ($\mu_X$). The gray areas represents the portions of the belief in which the true CATE is within one std from the CATE learned from ML-DS (orange band).
• Figure 4: Plot showing the correlation of variables $N_E$ and $\mu_A$ in the posterior beliefs of the agent after update with ML-DS information. The agent entertains multiple options
• Figure 5: Figures displaying the effects of different agent's priors when using ML-DS (in orange) and without it (in blue). In all plots, the x-axis represents the deviation of the agent's prior belief from the historical SCM's parameter for $\mu_Y$. The left plot displays the agent's prior and posterior (predictive) belief of CATE, while the right one shows the impact of the agent's decisions on the downstream outcome $Y$. In a rather wide band is the posterior within 1 std of the true mean (gray area).

Theorems & Definitions (7)

• Definition 2.1: Historical data generation
• Definition 2.2: Training data and prediction model
• Definition 2.3: Internal beliefs of the agent
• Definition 2.4: Agent's priors
• Definition 2.5: Agent belief update
• Definition 2.6: Agent model for decision making
• Definition 2.7: Decision rule