The Necessity of AI Audit Standards Boards

TL;DR

The paper argues that current efforts to create static AI auditing standards are problematic due to rapid technological evolution and diverse risk contexts. It proposes establishing an AI Audit Standards Board to develop and continuously update auditing methods, encompassing process, governance, and culture, rather than focusing solely on product-level evaluations. By drawing on safety-critical industries and applying a sociotechnical lens, it emphasizes multi-stakeholder engagement, ongoing monitoring, and application-specific risk considerations. The proposed governance structure aims to improve transparency, public trust, and safety, while avoiding regulatory overreach through a balance of principles and adaptable guidance.

Abstract

Auditing of AI systems is a promising way to understand and manage ethical problems and societal risks associated with contemporary AI systems, as well as some anticipated future risks. Efforts to develop standards for auditing Artificial Intelligence (AI) systems have therefore understandably gained momentum. However, we argue that creating auditing standards is not just insufficient, but actively harmful by proliferating unheeded and inconsistent standards, especially in light of the rapid evolution and ethical and safety challenges of AI. Instead, the paper proposes the establishment of an AI Audit Standards Board, responsible for developing and updating auditing methods and standards in line with the evolving nature of AI technologies. Such a body would ensure that auditing practices remain relevant, robust, and responsive to the rapid advancements in AI. The paper argues that such a governance structure would also be helpful for maintaining public trust in AI and for promoting a culture of safety and ethical responsibility within the AI industry. Throughout the paper, we draw parallels with other industries, including safety-critical industries like aviation and nuclear energy, as well as more prosaic ones such as financial accounting and pharmaceuticals. AI auditing should emulate those fields, and extend beyond technical assessments to include ethical considerations and stakeholder engagement, but we explain that this is not enough; emulating other fields' governance mechanisms for these processes, and for audit standards creation, is a necessity. We also emphasize the importance of auditing the entire development process of AI systems, not just the final products...