Policy alone is probably not the solution: A large-scale experiment on how developers struggle to design meaningful end-user explanations

TL;DR

The paper studies whether policy guidance and enforcement shape how developers design end-user explanations for an ML-powered medical device. It uses two large-scale controlled experiments with 194 participants to test policy variations in a diabetic retinopathy screening scenario. Results show policy language has limited impact on compliance and quality, while enforcement improves compliance modestly but fails to yield meaningful, end-user–oriented explanations. The authors highlight multiple failure modes and offer concrete recommendations for policymakers, educators, and tooling to advance toward more user-centered explanations beyond policy alone.

Abstract

Developers play a central role in determining how machine learning systems are explained in practice, yet they are rarely trained to design explanations for non-technical audiences. Despite this, transparency and explainability requirements are increasingly codified in regulation and organizational policy. It remains unclear how such policies influence developer behavior or the quality of the explanations they produce. We report results from two controlled experiments with 194 participants, typical developers without specialized training in human-centered explainable AI, who designed explanations for an ML-powered diabetic retinopathy screening tool. In the first experiment, differences in policy purpose and level of detail had little effect: policy guidance was often ignored and explanation quality remained low. In the second experiment, stronger enforcement increased formal compliance, but explanations largely remained poorly suited to medical professionals and patients. We further observed that across both experiments, developers repeatedly produced explanations that were technically flawed or difficult to interpret, framed for developers rather than end users, reliant on medical jargon, or insufficiently grounded in the clinical decision context and workflow, with developer-centric framing being the most prevalent. These findings suggest that policy and policy enforcement alone are insufficient to produce meaningful end-user explanations and that responsible AI frameworks may overestimate developers' ability to translate high-level requirements into human-centered designs without additional training, tools, or implementation support.

Figures (2)

• Figure 1: Examples of explanations in commercial products and student solutions for diabetic retinopathy diagnosis
• Figure 2: Our policy, highlighting the policy requirements selected for analysis (1--9)