Toward Third-Party Assurance of AI Systems: Design Requirements, Prototype, and Early Testing

TL;DR

The paper tackles the need for trustworthy AI evaluation by proposing a third-party AI assurance framework that covers both the design processes and the resulting outcomes across the AI lifecycle. It defines five design requirements (R1–R5), and presents a concrete prototype with four components—a Responsibility Assignment Matrix, an Interview Protocol, a Maturity Matrix, and an Assurance Report Template—validated through two real-world pilot applications and expert interviews. Early findings suggest the framework is necessary, sound, and usable, with evidence of identifying bespoke issues early in development, while also highlighting practical challenges such as the need for flexibility and resource considerations. This work lays the groundwork for standardized, external assurance practices in AI governance, with potential for broad adoption and future longitudinal validation to assess long-term impact on processes and outcomes.

Abstract

As Artificial Intelligence (AI) systems proliferate, the need for systematic, transparent, and actionable processes for evaluating them is growing. While many resources exist to support AI evaluation, they have several limitations. Few address both the process of designing, developing, and deploying an AI system and the outcomes it produces. Furthermore, few are end-to-end and operational, give actionable guidance, or present evidence of usability or effectiveness in practice. In this paper, we introduce a third-party AI assurance framework that addresses these gaps. We focus on third-party assurance to prevent conflict of interest and ensure credibility and accountability of the process. We begin by distinguishing assurance from audits in several key dimensions. Then, following design principles, we reflect on the shortcomings of existing resources to identify a set of design requirements for AI assurance. We then construct a prototype of an assurance process that consists of (1) a responsibility assignment matrix to determine the different levels of involvement each stakeholder has at each stage of the AI lifecycle, (2) an interview protocol for each stakeholder of an AI system, (3) a maturity matrix to assess AI systems' adherence to best practices, and (4) a template for an assurance report that draws from more mature assurance practices in business accounting. We conduct early validation of our AI assurance framework by applying the framework to two distinct AI use cases -- a business document tagging tool for downstream processing in a large private firm, and a housing resource allocation tool in a public agency -- and conducting expert validation interviews. Our findings show early evidence that our AI assurance framework is sound and comprehensive, usable across different organizational contexts, and effective at identifying bespoke issues with AI systems.

Figures (1)

• Figure 1: A flowchart of our AI assurance process. The figure specifies the resources that should be used in each step of the assurance process. The numbers correspond to the steps we describe in the main body of the paper (Section \ref{['sec:prototype']}).