Assessing Policy Updates: Toward Trust-Preserving Intelligent User Interfaces

TL;DR

Assessing model updates -- not just a single model -- is a critical design challenge for intelligent user interfaces and salient-contrast demonstrations significantly improved participants' ability to detect when updates helped or harmed performance, mitigating participants' bias towards assuming that feedback is always beneficial.

Abstract

Reinforcement learning agents are often updated with human feedback, yet such updates can be unreliable: reward misspecification, preference conflicts, or limited data may leave policies unchanged or even worse. Because policies are difficult to interpret directly, users face the challenge of deciding whether an update has truly helped. We propose that assessing model updates -- not just a single model -- is a critical design challenge for intelligent user interfaces. In a controlled study, participants provided feedback to an agent in a gridworld and then compared its original and updated policies. We evaluated four strategies for communicating updates: no demonstration, same-context, random-context, and salient-contrast demonstrations designed to highlight informative differences. Salient-contrast demonstrations significantly improved participants' ability to detect when updates helped or harmed performance, mitigating participants' bias towards assuming that feedback is always beneficial, and supported better trust calibration across contexts.

Figures (6)

• Figure 1: The feedback–assessment loop in interactive AI. A user observes an agent following policy $\pi_t$, provides feedback, and the agent updates to $\pi_{t+1}$. The system then presents a comparison of $\pi_t$ and $\pi_{t+1}$ using one of four strategies (control, same context, random context, or salient-contrast). The user assesses whether $V^H(\pi_{t+1})>V^H(\pi_t)$ and decides whether to adopt the new policy, informing (conceptual) trust/delegation and further interaction. Note: Trust/delegation was measured only after multiple rounds, not after each update.
• Figure 2: Example of an initial state of the environment used in the study.
• Figure 3: Side-by-side demonstration of the old policy and the updated policy during Stage 2.
• Figure 4: Correct agent choice across conditions. Two bar charts compare the four experimental groups. (a) Local correctness is computed on the feedback board, where the Same condition achieves the highest accuracy. (b) Generalized correctness is averaged across a fixed 18-board set under $\phi_T$, where Salient-contrast attains the highest scores. These results illustrate that same-context feedback supports local assessment, while salient-contrast demonstrations promote generalization. Error bars denote 95% confidence intervals.
• Figure 5: Correct agent choice by update direction (Same vs. Salient-contrast). Each panel shows the proportion of correct update evaluations split by whether the update improved or worsened performance. (a) On positive updates, the Same group achieves higher local correctness. (b) On negative updates, Salient-contrast shows substantially higher generalized correctness, indicating stronger ability to detect and reject harmful updates. Error bars denote 95% confidence intervals.