FactSheets: Increasing Trust in AI Services through Supplier's Declarations of Conformity

TL;DR

Fact Sheets for AI services address the trust gap by documenting intended use, performance, safety, security, and provenance of AI cloud services. The paper proposes a voluntary, standardized FactSheet inspired by supplier declarations of conformity and outlines a comprehensive item list and example implementations. It argues for multi-stakeholder standardization and discusses evolution toward third-party certification and automated, auditable reporting, including blockchain considerations. The contributions provide a foundation for transparent evaluation and safer, more accountable deployment of high-stakes AI services.

Abstract

Accuracy is an important concern for suppliers of artificial intelligence (AI) services, but considerations beyond accuracy, such as safety (which includes fairness and explainability), security, and provenance, are also critical elements to engender consumers' trust in a service. Many industries use transparent, standardized, but often not legally required documents called supplier's declarations of conformity (SDoCs) to describe the lineage of a product along with the safety and performance testing it has undergone. SDoCs may be considered multi-dimensional fact sheets that capture and quantify various aspects of the product and its development to make it worthy of consumers' trust. Inspired by this practice, we propose FactSheets to help increase trust in AI services. We envision such documents to contain purpose, performance, safety, security, and provenance information to be completed by AI service providers for examination by consumers. We suggest a comprehensive set of declaration items tailored to AI and provide examples for two fictitious AI services in the appendix of the paper.