Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-Service or Not? How to Guide Practitioners in Classifying AI Systems Under the EU AI Act

Ronald Schnitzer, Maximilian Hoeving, Sonja Zillner

TL;DR

Evaluating how industrial practitioners apply the Risk Classification Scheme using a self-service, web-based decision-support tool highlights critical challenges in interpreting legal definitions and regulatory scope, and shows that targeted support can significantly enhance the risk classification process.

Abstract

In August 2024, the EU Artificial Intelligence Act (AIA) came into force, marking the world's first large-scale regulatory framework for AI. Central to the AIA is a risk-based approach, aligning regulatory obligations with the potential harm posed by AI systems. To operationalize this, the AIA defines a Risk Classification Scheme (RCS), categorizing systems into four levels of risk. While this aligns with the theoretical foundations of risk-based regulations, the practical application of the RCS is complex and requires expertise across legal, technical, and domain-specific areas. Despite increasing academic discussion, little empirical research has explored how practitioners apply the RCS in real-world contexts. This study addresses this gap by evaluating how industrial practitioners apply the RCS using a self-service, web-based decision-support tool. Following a Design Science Research (DSR) approach, two evaluation phases involving 78 practitioners across diverse domains were conducted. Our findings highlight critical challenges in interpreting legal definitions and regulatory scope, and show that targeted support, such as clear explanations and practical examples, can significantly enhance the risk classification process. The study provides actionable insights for tool designers and policymakers aiming to support AIA compliance in practice.

Self-Service or Not? How to Guide Practitioners in Classifying AI Systems Under the EU AI Act

TL;DR

Evaluating how industrial practitioners apply the Risk Classification Scheme using a self-service, web-based decision-support tool highlights critical challenges in interpreting legal definitions and regulatory scope, and shows that targeted support can significantly enhance the risk classification process.

Abstract

In August 2024, the EU Artificial Intelligence Act (AIA) came into force, marking the world's first large-scale regulatory framework for AI. Central to the AIA is a risk-based approach, aligning regulatory obligations with the potential harm posed by AI systems. To operationalize this, the AIA defines a Risk Classification Scheme (RCS), categorizing systems into four levels of risk. While this aligns with the theoretical foundations of risk-based regulations, the practical application of the RCS is complex and requires expertise across legal, technical, and domain-specific areas. Despite increasing academic discussion, little empirical research has explored how practitioners apply the RCS in real-world contexts. This study addresses this gap by evaluating how industrial practitioners apply the RCS using a self-service, web-based decision-support tool. Following a Design Science Research (DSR) approach, two evaluation phases involving 78 practitioners across diverse domains were conducted. Our findings highlight critical challenges in interpreting legal definitions and regulatory scope, and show that targeted support, such as clear explanations and practical examples, can significantly enhance the risk classification process. The study provides actionable insights for tool designers and policymakers aiming to support AIA compliance in practice.
Paper Structure (28 sections, 5 figures, 3 tables)

This paper contains 28 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Risk-based regulation
  4. The European AI Act
  5. Software Tooling for Legal Applications
  6. Implementations of the RCS
  7. Methodology
  8. Artifact Design
  9. Evaluation Design
  10. Artifact Beta Evaluation Phase
  11. Final Evaluation Phase
  12. Evaluation Results
  13. Qualitative results from the artifact beta evaluation phase
  14. Unclear terms and definitions in the RCS hinder consistent understanding
  15. Accurate classification requires detailed know-how on European Union harmonisation legislations
  16. ...and 13 more sections

Figures (5)

  • Figure 1: Research process of this study with inputs and outputs shown at the top. Below, each step is mapped to the DSR process peffers_design_2007, and the related section in this paper.
  • Figure 2: Decision-Tree representation of the implemented RCS.
  • Figure 3: Interpolated Median lorenz_ranking_2018 and Percent Favourable nulty_adequacy_2008 evaluated for all statements in the participant cross-evaluation (Ex_beta).
  • Figure 4: Distribution of accesses to supplementary materials per user, grouped by types of supplementary material. The title of each subplot denotes the percentage of users that accessed the respective supporting material at least once throughout their use.
  • Figure 5: Interpolated Median lorenz_ranking_2018 and Percent Favourable nulty_adequacy_2008 evaluated for all statements in the final evaluation Likert-Scale survey (Ex_final).