Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Explanation: Evidentiary Rights for Algorithmic Accountability

Matthew Stewart

Abstract

Algorithmic accountability scholarship has focused heavily on explanation, helping affected parties understand why decisions were made. We argue this focus is insufficient. Explanation without evidentiary access does not enable meaningful contestation. A person told "your risk score was 0.73" understands the decision but cannot verify the score, test alternatives, or produce counter-evidence. We introduce a taxonomy of contestation failures, showing that most accountability interventions address only one failure mode (opacity) while leaving four others unaddressed. Drawing on analysis of 168 legal cases spanning algorithmic decision-making contexts, we find that contestation faces a two-gate structure: a procedural gate (evidentiary access) and a doctrinal gate (substantive liability rules). Among litigated cases, those without evidence access almost never succeed (9%); those with access succeed at rates approaching 97% in domains without liability shields. Where doctrinal immunities apply (e.g., Section 230), even full evidentiary scrutiny produces no liability. This association almost certainly reflects selection effects; our empirical contribution is diagnostic rather than causal. The data identify where contestation fails among observable cases, not whether providing access would change outcomes for currently-excluded cases. We propose evidentiary rights as the missing procedural component, and develop counterfactual interrogation rights that allow affected parties to probe decision systems with modified inputs and observe whether outcomes change, without requiring disclosure of model internals. This reframes algorithmic accountability from a transparency problem to a procedural rights problem.

Beyond Explanation: Evidentiary Rights for Algorithmic Accountability

Abstract

Algorithmic accountability scholarship has focused heavily on explanation, helping affected parties understand why decisions were made. We argue this focus is insufficient. Explanation without evidentiary access does not enable meaningful contestation. A person told "your risk score was 0.73" understands the decision but cannot verify the score, test alternatives, or produce counter-evidence. We introduce a taxonomy of contestation failures, showing that most accountability interventions address only one failure mode (opacity) while leaving four others unaddressed. Drawing on analysis of 168 legal cases spanning algorithmic decision-making contexts, we find that contestation faces a two-gate structure: a procedural gate (evidentiary access) and a doctrinal gate (substantive liability rules). Among litigated cases, those without evidence access almost never succeed (9%); those with access succeed at rates approaching 97% in domains without liability shields. Where doctrinal immunities apply (e.g., Section 230), even full evidentiary scrutiny produces no liability. This association almost certainly reflects selection effects; our empirical contribution is diagnostic rather than causal. The data identify where contestation fails among observable cases, not whether providing access would change outcomes for currently-excluded cases. We propose evidentiary rights as the missing procedural component, and develop counterfactual interrogation rights that allow affected parties to probe decision systems with modified inputs and observe whether outcomes change, without requiring disclosure of model internals. This reframes algorithmic accountability from a transparency problem to a procedural rights problem.
Paper Structure (42 sections, 4 figures, 18 tables)

This paper contains 42 sections, 4 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. A Taxonomy of Contestation Failures
  4. Evidence Access and Accountability Outcomes
  5. Results
  6. Interpreting the Pattern
  7. Testing the Core Claim: Explanation Quality vs. Evidence Access
  8. What Evidence Access Provides
  9. Why Existing Frameworks Fail
  10. Evidentiary Rights: Counterfactual Interrogation
  11. Implementation Pathways
  12. Limitations
  13. Conclusion
  14. Research Agenda for Types 3--5
  15. Implementation for Low-Resource Populations
  16. ...and 27 more sections

Figures (4)

  • Figure 1: Taxonomy of contestation failures: conceptual framework and observed distribution. (a) The contestation pipeline shows where barriers occur; current frameworks address opacity (Type 1) but leave evidentiary access (Type 2) undertheorized. (b) Among cases reaching litigation, evidence access is the most frequently observed barrier: 126 of 168 cases (74%) involve evidentiary inaccessibility as the primary obstacle. This distribution reflects litigation selection patterns, not necessarily the prevalence of barriers overall---Type 1 failures may not generate litigation, while Types 3--5 failures may be harder to identify in court records.
  • Figure 2: Case distribution by domain and evidence access level (n=168). Blue bars: cases with evidence access (Full or Partial); orange bars: cases without access. Platform liability cases had full access yet failed due to Section 230 doctrinal immunity.
  • Figure 3: Counterfactual interrogation process flow. Steps 1--4 are sequential: adverse decision received (1), portal access granted (2), perturbations submitted (3), output changes observed (4). A divergence assessment (diamond) then branches: significant divergence leads to filing an appeal with evidence (5) and independent adjudication (6); no divergence allows retesting with different factors.
  • Figure 4: PRISMA-style case identification flow diagram. Records were identified through legal database searches (Westlaw, LexisNexis), regulatory enforcement databases (EEOC, CFPB, FTC, DOJ), and secondary sources (academic surveys, investigative journalism, advocacy databases). International cases were drawn from European court databases, data protection authority decisions, and comparative law sources.