Legacy Procurement Practices Shape How U.S. Cities Govern AI: Understanding Government Employees' Practices, Challenges, and Needs

TL;DR

The paper investigates how legacy public procurement practices shape the governance of AI in U.S. cities, revealing that decades-old laws and norms structure which AI tools are adopted, who has oversight, and how harms are addressed. Through 19 semi-structured interviews across 7 cities, it demonstrates that formal procurement processes and cost-based thresholds often hinder proactive addressing of AI harms, while emerging practices such as AI-specific risk assessments and vendor-agnostic templates are being adopted selectively. The study identifies three key challenges—information asymmetry with vendors, limited leverage over vendors, and unclear sharing of ongoing governance—to guide future reform. It discusses implications for the FAccT community to develop tools, standards, and collaborative frameworks that bolster accountability, transparency, and citizen protections in public AI procurement.

Abstract

Most AI tools adopted by governments are not developed internally, but instead are acquired from third-party vendors in a process called public procurement. In this paper, we conduct the first empirical study of how United States cities' procurement practices shape critical decisions surrounding public sector AI. We conduct semi-structured interviews with 19 city employees who oversee AI procurement across 7 U.S. cities. We found that cities' legacy procurement practices, which are shaped by decades-old laws and norms, establish infrastructure that determines which AI is purchased, and which actors hold decision-making power over procured AI. We characterize the emerging actions cities have taken to adapt their purchasing practices to address algorithmic harms. From employees' reflections on real-world AI procurements, we identify three key challenges that motivate but are not fully addressed by existing AI procurement reform initiatives. Based on these findings, we discuss implications and opportunities for the FAccT community to support cities in foreseeing and preventing AI harms throughout the public procurement processes.

Figures (1)

• Figure 1: Six common steps that occurred in cities' public procurement processes. While purchases that went through a full solicitation (e.g., an RFP) proceeded linearly through these steps, as shown by the long blue arrow, AI acquired via alternative purchasing pathways such as cooperative purchasing agreements ("piggyback contracts") or under cost thresholds (e.g., using purchasing cards) skipped past several steps, illustrated by the short blue arrows. In our paper, we discuss meaningful differences across cities' processes.