Algorithmic Assistance with Recommendation-Dependent Preferences

TL;DR

The paper studies how algorithmic recommendations can influence decisions not only through information but by shifting decision-makers' preferences via reference points and defaults. It develops a principal–agent model where a designer's recommendations affect actions, formalizes loss with Δ_I and Δ_{II} penalties, and shows that recommendation dependence can reduce efficiency. The authors derive minimax and triage-type strategies that strategically withhold or neutralize recommendations to improve outcomes, and they extend the framework to continuous risk scores and implicit recommendations. The work provides design principles for improving human–AI collaboration in high-stakes settings by accounting for behavioral responses to algorithmic advice and offering practical, data-efficient approaches.

Abstract

When an algorithm provides risk assessments, we typically think of them as helpful inputs to human decisions, such as when risk scores are presented to judges or doctors. However, a decision-maker may react not only to the information provided by the algorithm. The decision-maker may also view the algorithmic recommendation as a default action, making it costly for them to deviate, such as when a judge is reluctant to overrule a high-risk assessment for a defendant or a doctor fears the consequences of deviating from recommended procedures. To address such unintended consequences of algorithmic assistance, we propose a model of joint human-machine decision-making. Within this model, we consider the effect and design of algorithmic recommendations when they affect choices not just by shifting beliefs, but also by altering preferences. We motivate this assumption from institutional factors, such as a desire to avoid audits, as well as from well-established models in behavioral science that predict loss aversion relative to a reference point. We show that recommendation-dependent preferences create inefficiencies where the decision-maker is overly responsive to the recommendation. As a remedy, we discuss algorithms that strategically withhold recommendations and show how they can improve the quality of final decisions. Concretely, we prove that an intuitive algorithm achieves minimax optimality by sending recommendations only when it is confident that their implementation would improve over an unassisted baseline decision.

Figures (6)

• Figure 1: Joint distribution of outcome, human signal, and machine signal in \ref{['exm1']}, along with the optimal decision of a human decision-maker acting without recommendation.
• Figure 2: Comparison of machine-assisted decisions without recommendation dependence (left) and with recommendation dependence (right) for \ref{['exm1']}.
• Figure 3: Optimal decision thresholds are adjusted in response to recommendation dependence. The thin lines show the thresholds in \ref{['fig:ex-advised']}, while the arrows depict the optimal change in machine threshold (left) and resulting adjustment of conditional decision-maker choices (right).
• Figure 4: Incorporating a neutral recommendation provides additional information to the human decision-maker, while also limiting the region in which recommendation dependence distorts choices.
• Figure 5: Illustration of the minimax optimal recommendation for $p^* = .4$ (left) and the thresholds at which recommendations are sent as a function of $p^*$ (right) for \ref{['exm1']}.

Theorems & Definitions (52)

• Proposition 1: Derivation from loss aversion
• Proposition 2: Costly defaults as recommendation-dependent preferences
• Proposition 3: Maximin optimal audits
• Remark 1: Recommendation-dependent thresholds
• Remark 2: Recommendation dependence increases adherence and inefficiency
• Proposition 4: More information does not imply better performance
• Example 1: name=Independent uniform signals,label=exm1
• Remark 3: Improvement over the machine
• Proposition 5: Inefficient recommendations
• Example 2: name=Independent uniform signals,continues=exm1