AI Procurement Checklists: Revisiting Implementation in the Age of AI Governance

TL;DR

The paper addresses the challenge of regulating AI use in government given tensions between ethical safeguards and procurement efficiency. It analyzes two concrete checklists, CDADM and WEF AI Procurement in a Box, through interviews with officials in Brazil, Singapore, and Canada. It identifies three key pitfalls: need for AI expertise, procurement loopholes, and insufficient transparency, and offers recommendations including scaling expert integration, closing loopholes, and enhancing transparency and accountability. The study suggests systemic interventions like cross-sector partnerships and standardized private-sector AI audits to improve governance and public trust. The practical impact is to guide US regulatory efforts by learning from mature regimes.

Abstract

Public sector use of AI has been quietly on the rise for the past decade, but only recently have efforts to regulate it entered the cultural zeitgeist. While simple to articulate, promoting ethical and effective roll outs of AI systems in government is a notoriously elusive task. On the one hand there are hard-to-address pitfalls associated with AI-based tools, including concerns about bias towards marginalized communities, safety, and gameability. On the other, there is pressure not to make it too difficult to adopt AI, especially in the public sector which typically has fewer resources than the private sector$\unicode{x2014}$conserving scarce government resources is often the draw of using AI-based tools in the first place. These tensions create a real risk that procedures built to ensure marginalized groups are not hurt by government use of AI will, in practice, be performative and ineffective. To inform the latest wave of regulatory efforts in the United States, we look to jurisdictions with mature regulations around government AI use. We report on lessons learned by officials in Brazil, Singapore and Canada, who have collectively implemented risk categories, disclosure requirements and assessments into the way they procure AI tools. In particular, we investigate two implemented checklists: the Canadian Directive on Automated Decision-Making (CDADM) and the World Economic Forum's AI Procurement in a Box (WEF). We detail three key pitfalls around expertise, risk frameworks and transparency, that can decrease the efficacy of regulations aimed at government AI use and suggest avenues for improvement.