A moving target in AI-assisted decision-making: Dataset shift, model updating, and the problem of update opacity
Joshua Hatherley
TL;DR
The paper addresses how dataset shift undermines the long‑term reliability of AI-assisted decision‑making and introduces update opacity as a new epistemic/safety challenge that arises when updating changes system reasoning. It surveys conventional explainability and reporting approaches and proposes targeted alternatives (bi‑factual explanations, dynamic reporting, and update compatibility), but finds that none fully resolves the opacity introduced by updates. The analysis emphasizes that the human‑AI team's performance hinges on understanding how updates affect outputs, not just isolated improvements in accuracy. The work highlights the importance of interpretable, update‑aware designs and calls for further research into epistemic safeguards and safer update practices in high‑stakes contexts.
Abstract
Machine learning (ML) systems are vulnerable to performance decline over time due to dataset shift. To address this problem, experts often suggest that ML systems should be regularly updated to ensure ongoing performance stability. Some scholarly literature has begun to address the epistemic and ethical challenges associated with different updating methodologies. Thus far, however, little attention has been paid to the impact of model updating on the ML-assisted decision-making process itself, particularly in the AI ethics and AI epistemology literatures. This article aims to address this gap in the literature. It argues that model updating introduces a new sub-type of opacity into ML-assisted decision-making -- update opacity -- that occurs when users cannot understand how or why an update has changed the reasoning or behaviour of an ML system. This type of opacity presents a variety of distinctive epistemic and safety concerns that available solutions to the black box problem in ML are largely ill-equipped to address. A variety of alternative strategies may be developed or pursued to address the problem of update opacity more directly, including bi-factual explanations, dynamic model reporting, and update compatibility. However, each of these strategies presents its own risks or carries significant limitations. Further research will be needed to address the epistemic and safety concerns associated with model updating and update opacity going forward.