A Q-learning Approach for Adherence-Aware Recommendations

TL;DR

This work tackles high-stakes decision settings where a human decision-maker follows AI recommendations with imperfect adherence. It introduces adherence-aware Q-learning, which online-estimates the adherence level $\theta$ and updates the Q-function by blending the future values from the recommended action and the baseline action, yielding a policy that optimizes the expected discounted reward. A contraction-based convergence proof guarantees a unique fixed point $Q^{\infty}=Q^*$, and the method is validated on inventory and machine-replacement MDPs, showing faster convergence and higher actual rewards than baselines and standard Q-learning. The results support real-time, adherence-aware policy learning in expert-in-the-loop systems and point to future work on learning baselines, partial observability, and decentralized extensions.

Abstract

In many real-world scenarios involving high-stakes and safety implications, a human decision-maker (HDM) may receive recommendations from an artificial intelligence while holding the ultimate responsibility of making decisions. In this letter, we develop an "adherence-aware Q-learning" algorithm to address this problem. The algorithm learns the "adherence level" that captures the frequency with which an HDM follows the recommended actions and derives the best recommendation policy in real time. We prove the convergence of the proposed Q-learning algorithm to the optimal value and evaluate its performance across various scenarios.

Figures (4)

• Figure 1: Process of the actual law implementation.
• Figure 2: Transition probabilities for repair (left) and wait (right).
• Figure 3: Inventory control results.
• Figure 4: Machine replacement results.

Theorems & Definitions (8)

• Remark 1
• Definition 1
• Lemma 1
• proof
• Lemma 2
• proof
• Theorem 1
• proof