Model Cards Revisited: Bridging the Gap Between Theory and Practice for Ethical AI Requirements
Tim Puhlfürß, Julia Butzke, Walid Maalej
TL;DR
This paper investigates how ethical AI requirements articulated in guidelines and documentation frameworks align with real-world documentation artifacts. It conducts a thematic analysis of 83 guidelines, three documentation frameworks, three empirical studies, and selected model-card artifacts to derive a taxonomy of 43 requirements organized into four principles. The findings show that current practice prioritizes capabilities and reliability while underreporting explainability, user autonomy, fairness, and risk management, revealing a gap between theory and practice. The proposed taxonomy provides a foundation for a revised, holistic, and machine-readable documentation framework to guide practitioners, auditors, and platforms in responsible AI documentation.
Abstract
Model cards are the primary documentation framework for developers of artificial intelligence (AI) models to communicate critical information to their users. Those users are often developers themselves looking for relevant documentation to ensure that their AI systems comply with the ethical requirements of existing laws, guidelines, and standards. Recent studies indicate inadequate model documentation practices, suggesting a gap between AI requirements and current practices in model documentation. To understand this gap and provide actionable guidance to bridge it, we conducted a thematic analysis of 26 guidelines on ethics and AI, three AI documentation frameworks, three quantitative studies of model cards, and ten actual model cards. We identified a total of 43 ethical requirements relevant to model documentation and organized them into a taxonomy featuring four themes and twelve sub-themes representing ethical principles. Our findings indicate that model developers predominantly emphasize model capabilities and reliability in the documentation while overlooking other ethical aspects, such as explainability, user autonomy, and fairness. This underscores the need for enhanced support in documenting ethical AI considerations. Our taxonomy serves as a foundation for a revised model card framework that holistically addresses ethical AI requirements.