Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards a Non-Ideal Methodological Framework for Responsible ML

Ramaravind Kommiya Mothilal, Shion Guha, Syed Ishtiaque Ahmed

TL;DR

The paper tackles the lack of a concrete methodological framework for responsible ML among technically oriented practitioners. It uses a qualitative study of 22 ML practitioners across domains, analyzed through ideal and non-ideal theoretical lenses, to show that practice sits on a spectrum from idealized structuring to non-ideal, under-documented approaches. The authors propose a non-ideal theory–inspired framework that documents assumptions, maps imperfect data properties to abstract values and concrete interventions, and supports collaboration with diverse stakeholders. This framework aims to provide a public-facing accountability tool and structured guidance to improve value alignment and reflexivity throughout the ML lifecycle. The work highlights the need for systematic documentation and iterative mapping to better address real-world complexities in responsible ML.

Abstract

Though ML practitioners increasingly employ various Responsible ML (RML) strategies, their methodological approach in practice is still unclear. In particular, the constraints, assumptions, and choices of practitioners with technical duties -- such as developers, engineers, and data scientists -- are often implicit, subtle, and under-scrutinized in HCI and related fields. We interviewed 22 technically oriented ML practitioners across seven domains to understand the characteristics of their methodological approaches to RML through the lens of ideal and non-ideal theorizing of fairness. We find that practitioners' methodological approaches fall along a spectrum of idealization. While they structured their approaches through ideal theorizing, such as by abstracting RML workflow from the inquiry of applicability of ML, they did not pay deliberate attention and systematically documented their non-ideal approaches, such as diagnosing imperfect conditions. We end our paper with a discussion of a new methodological approach, inspired by elements of non-ideal theory, to structure technical practitioners' RML process and facilitate collaboration with other stakeholders.

Towards a Non-Ideal Methodological Framework for Responsible ML

TL;DR

The paper tackles the lack of a concrete methodological framework for responsible ML among technically oriented practitioners. It uses a qualitative study of 22 ML practitioners across domains, analyzed through ideal and non-ideal theoretical lenses, to show that practice sits on a spectrum from idealized structuring to non-ideal, under-documented approaches. The authors propose a non-ideal theory–inspired framework that documents assumptions, maps imperfect data properties to abstract values and concrete interventions, and supports collaboration with diverse stakeholders. This framework aims to provide a public-facing accountability tool and structured guidance to improve value alignment and reflexivity throughout the ML lifecycle. The work highlights the need for systematic documentation and iterative mapping to better address real-world complexities in responsible ML.

Abstract

Though ML practitioners increasingly employ various Responsible ML (RML) strategies, their methodological approach in practice is still unclear. In particular, the constraints, assumptions, and choices of practitioners with technical duties -- such as developers, engineers, and data scientists -- are often implicit, subtle, and under-scrutinized in HCI and related fields. We interviewed 22 technically oriented ML practitioners across seven domains to understand the characteristics of their methodological approaches to RML through the lens of ideal and non-ideal theorizing of fairness. We find that practitioners' methodological approaches fall along a spectrum of idealization. While they structured their approaches through ideal theorizing, such as by abstracting RML workflow from the inquiry of applicability of ML, they did not pay deliberate attention and systematically documented their non-ideal approaches, such as diagnosing imperfect conditions. We end our paper with a discussion of a new methodological approach, inspired by elements of non-ideal theory, to structure technical practitioners' RML process and facilitate collaboration with other stakeholders.
Paper Structure (15 sections, 1 figure, 1 table)

This paper contains 15 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Related Literature
  3. Responsible ML in Practice
  4. Theorizing Responsible ML Practices
  5. Methods
  6. Findings
  7. Constraints Determining Methodological Choices
  8. Defining Characteristics of RML Methodology
  9. Structuring through Ideal Theorizing
  10. Not Documenting non-ideal Approaches
  11. Supporting by Mapping Values to Interventions
  12. Discussion
  13. Components of Mapping
  14. Procedure of Mapping
  15. Practical Considerations and Limitations

Figures (1)

  • Figure 1: An imagination of how our methodological framework can be presented. To illustrate, consider 'Feature dependencies' as an undesirable property mapped to the realized goal, 'Robust to high-cardinal (categorical) data.' This implies feature dependencies affect achieving the goal of building robust models with high-cardinal categorical features. Note that such visualizations can help identify and keep track of unaddressed undesirable properties (such as "feature dependencies" and "incomplete operationalization of larger focus of ML systems" in this example) and incomplete mappings between different components (for instance, "feature dependencies" can affect addressing distribution shifts but are not mapped explicitly in this example).