Justified Evidence Collection for Argument-based AI Fairness Assurance

TL;DR

This work addresses the challenge of ensuring fairness in AI-enabled systems by proposing a dynamic, argument-based assurance framework that integrates justification-based reasoning with continuous evidence collection. It introduces a modular TEA-based platform for creating structured assurance cases and employing transparency artefacts (data cards, model cards, use case cards) to document and monitor fairness throughout the lifecycle. A two-stage process—requirements governance and ongoing evidence gathering—supports proactive fairness monitoring and aligns with risk-management practices, demonstrated through a finance-focused heuristic case study. The approach advances justified trust in AI by making assumptions explicit, enabling multi-stakeholder deliberation, and providing open-source tooling to operationalize fairness governance across domains and maturity levels.

Abstract

It is well recognised that ensuring fair AI systems is a complex sociotechnical challenge, which requires careful deliberation and continuous oversight across all stages of a system's lifecycle, from defining requirements to model deployment and deprovisioning. Dynamic argument-based assurance cases, which present structured arguments supported by evidence, have emerged as a systematic approach to evaluating and mitigating safety risks and hazards in AI-enabled system development and have also been extended to deal with broader normative goals such as fairness and explainability. This paper introduces a systems-engineering-driven framework, supported by software tooling, to operationalise a dynamic approach to argument-based assurance in two stages. In the first stage, during the requirements planning phase, a multi-disciplinary and multi-stakeholder team define goals and claims to be established (and evidenced) by conducting a comprehensive fairness governance process. In the second stage, a continuous monitoring interface gathers evidence from existing artefacts (e.g. metrics from automated tests), such as model, data, and use case documentation, to support these arguments dynamically. The framework's effectiveness is demonstrated through an illustrative case study in finance, with a focus on supporting fairness-related arguments.

Figures (6)

• Figure 1: A toy model showing an incomplete example of an assurance case for a fairness-related goal.
• Figure 2: From structured arguments to collecting evidence: In the argument-based assurance process, we defined property claims using a hierarchical mechanism based on ML development stages and components. This figure illustrates this hierarchical process and points respective transparency artefacts for evidence collection.
• Figure 3: A simplified example use case card for a financial sentiment analysis system. The template is obtained from hupont_use_2023.
• Figure 4: Creating arguments to achieve a fairer financial sentiment analysis system.
• Figure 5: Categorical diagram of selected evaluation and mitigation techniques that can address fairness issues directly in a component-based risk analysis.