The Value of AI Advice: Personalized and Value-Maximizing AI Advisors Are Necessary to Reliably Benefit Experts and Organizations
Nicholas Wolczynski, Maytal Saar-Tsechansky, Tong Wang
TL;DR
The paper confronts the mismatch between AI advisor performance and real-world value in high-stakes decisions, arguing that reliability and value-maximization require accounting for advising costs and human behavior. It introduces the ReV-AI framework and a concrete, interpretable implementation called TeamRules (TR), designed to learn context-specific, selective advice with inherent persuasiveness. Through synthetic and empirical analyses across multiple domains and decision-maker behaviors, TR consistently adds more value than traditional task-focused or non-personalized advisors, especially under costly or miscalibrated human advice-taking. The work demonstrates that superhuman AI accuracy is neither necessary nor sufficient for value, and that value-driven, customizable advisors can substantially improve outcomes while mitigating potential harms. These findings carry important managerial implications for deploying AI advisers in real organizations and lay groundwork for future extensions to broader HCI configurations and decision domains.
Abstract
Despite advances in AI's performance and interpretability, AI advisors can undermine experts' decisions and increase the time and effort experts must invest to make decisions. Consequently, AI systems deployed in high-stakes settings often fail to consistently add value across experts and organizations and can even diminish the value that experts alone provide. Beyond harm in specific domains, such outcomes impede progress in research and practice, underscoring the need to understand when and why different AI advisors add or diminish value. To bridge this gap, we stress the importance of assessing the value AI advice brings to real-world contexts when designing and evaluating AI advisors. Building on this perspective, we characterize key pillars -- pathways through which AI advice impacts value -- and develop a framework that incorporates these pillars to create reliable, personalized, and value-adding advisors. Our results highlight the need for value-driven development of AI advisors that advise selectively, are tailored to experts' unique behaviors, and are optimized for context-specific trade-offs between decision improvements and advising costs. They also reveal how the lack of inclusion of these pillars in the design of AI advising systems may be contributing to the failures observed in practical applications.