Investigating the Capabilities and Limitations of Machine Learning for Identifying Bias in English Language Data with Information and Heritage Professionals

TL;DR

This paper reframes ML bias as a phenomenon to be identified in data rather than eradicated, focusing on gender bias in GLAM catalog descriptions. It combines supervised ML models (LC, PNOC, OSC) trained on a human-annotated dataset with a participatory workshop of information and heritage professionals to assess conceptualizations of bias and the utility of ML outputs. Findings reveal bias is contextual, dynamic, and intertwined with power structures, highlighting limitations of purely automated bias mitigation and underscoring the value of human-centered, mixed-methods evaluation. The work offers a transferable workflow that uses ML to surface bias while enabling critical discission and responsible management in GLAM, with implications for broader ML bias/fairness research and practice.

Abstract

Despite numerous efforts to mitigate their biases, ML systems continue to harm already-marginalized people. While predominant ML approaches assume bias can be removed and fair models can be created, we show that these are not always possible, nor desirable, goals. We reframe the problem of ML bias by creating models to identify biased language, drawing attention to a dataset's biases rather than trying to remove them. Then, through a workshop, we evaluated the models for a specific use case: workflows of information and heritage professionals. Our findings demonstrate the limitations of ML for identifying bias due to its contextual nature, the way in which approaches to mitigating it can simultaneously privilege and oppress different communities, and its inevitability. We demonstrate the need to expand ML approaches to bias and fairness, providing a mixed-methods approach to investigating the feasibility of removing bias or achieving fairness in a given ML use case.

Figures (5)

• Figure 1: A flow chart of our four-step research process, with each step in boldface and the main tasks for the steps radiating out and above each step. This paper summarizes the entire research process while focusing on the last step, "Evaluate the classification models," and the substep "Workshop," through which we qualitatively evaluate the models (§\ref{['workshop']}). Previous publications, as referenced in the flow chart and throughout this paper, provide more detail on the steps for defining gendered and gender biased language Havens_2020, creating the annotated dataset Havens_2022, and training and quantitatively evaluating the classification models Havens_2024.
• Figure 2: The front of the worksheet participants were given for the workshop's first activity displaying the coding taxonomy that outlines how the ML models were trained to classify descriptions.
• Figure 4: The front of of the worksheet participants were given for the workshop's second activity. The tables at the top show total quantities of Stereotype and Omission codes in the collections ("fonds") with the highest total counts of those codes. The tables at the bottom show the most common languages of the archival records in the collections listed in the tables above.
• Figure 5: The back of of the worksheet participants were given for the workshop's second activity displaying two charts with quantities of taxonomy codes across the entire training dataset, and one chart visualizing the performance of a classifier as represented by true positive, false positive, true negative, and false negative counts.
• Figure 6: The same flow chart of our four-step research process as Figure \ref{['fig:research-process']} with three dotted line arrows added. These new arrows connect the evaluation step to each of the previous steps, indicating the iterations that future work can follow to refine how a model's task is defined or conceptualized, refine the contents and quality of a dataset, and refine the design of a model. Using traditional ML approaches enables these iterations to be undertaken quickly and at low cost. Once the quantitative and qualitative evaluations of a model are satisfactory, deep learning approaches could be used to create a model on the iteratively refined training dataset.