The Balancing Act of Policies in Developing Machine Learning Explanations

TL;DR

The paper investigates how policy design affects ML explanations by conducting a classroom experiment with six policy conditions that vary in length and purpose. The approach combines global and local explanations for an ML-based diabetic retinopathy screening task and uses qualitative coding to assess compliance. The findings show that policy length subtly boosts engagement with some requirements, but policy purpose has no measurable effect, and overall explanations are poor and rarely reflect stakeholder priorities. The work underscores the difficulty of achieving meaningful transparency through policy alone and advocates for stakeholder-centered policy design to improve explainability in practice.

Abstract

Machine learning models are often criticized as opaque from a lack of transparency in their decision-making process. This study examines how policy design impacts the quality of explanations in ML models. We conducted a classroom experiment with 124 participants and analyzed the effects of policy length and purpose on developer compliance with policy requirements. Our results indicate that while policy length affects engagement with some requirements, policy purpose has no effect, and explanation quality is generally poor. These findings highlight the challenge of effective policy development and the importance of addressing diverse stakeholder perspectives within explanations.

Figures (2)

• Figure 1: Local Explanation Examples: The Anchor image is meaningless to stakeholders, while the integrated gradient is more accessible to trained users.
• Figure 2: Participant Self-Evaluated Compliance by Policy Length