Secure human oversight of AI: Threat modeling in a socio-technical context

TL;DR

It is argued that human oversight creates a new attack surface within the safety, security, and accountability architecture of AI operations, and model human oversight as an IT application for the purpose of systematic threat modeling of the human oversight process.

Abstract

Human oversight of AI is promoted as a safeguard against risks such as inaccurate outputs, system malfunctions, or violations of fundamental rights, and is mandated in regulation like the European AI Act. Yet debates on human oversight have largely focused on its effectiveness, while overlooking a critical dimension: the security of human oversight. We argue that human oversight creates a new attack surface within the safety, security, and accountability architecture of AI operations. Drawing on cybersecurity perspectives, we model human oversight as an IT application for the purpose of systematic threat modeling of the human oversight process. Threat modeling allows us to identify security risks within human oversight and points towards possible mitigation strategies. Our contributions are: (1) introducing a security perspective on human oversight, (2) offering researchers and practitioners guidance on how to approach their human oversight applications from a security point of view, and (3) providing a systematic overview of attack vectors and hardening strategies to enable secure human oversight of AI.

Figures (1)

• Figure 1: Data flow diagram (DFD) of the abstracted human oversight application used for threat modeling. Arrows indicate the directed paths of data through the application. Each arrow is labeled with the type of data that is transported. Circles indicate processes, two concentric circles indicate multiple subprocesses, and rectangles indicate external entities. Privilege boundaries are drawn with red dashed lines. HO = Human Oversight.