COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Act

TL;DR

This work introduces COMPL-AI, a technical interpretation of the EU AI Act tailored for LLMs, and an open-source Act-centered benchmarking suite that translates regulatory requirements into concrete, measurable benchmarks. By evaluating 12 LLMs across robustness, safety, privacy, transparency, fairness, and environmental impact, the authors reveal systemic gaps in current models and evaluation practices, emphasizing that capability-focused benchmarks alone are insufficient for compliance. The framework demonstrates the feasibility and challenges of translating regulatory obligations into actionable technical tests, and it highlights the need for standardization, broader reporting, and development of regulation-aligned benchmarks—paving the way for Code of Practice efforts and more comprehensive regulatory conformance in practice.

Abstract

The EU's Artificial Intelligence Act (AI Act) is a significant step towards responsible AI development, but lacks clear technical interpretation, making it difficult to assess models' compliance. This work presents COMPL-AI, a comprehensive framework consisting of (i) the first technical interpretation of the EU AI Act, translating its broad regulatory requirements into measurable technical requirements, with the focus on large language models (LLMs), and (ii) an open-source Act-centered benchmarking suite, based on thorough surveying and implementation of state-of-the-art LLM benchmarks. By evaluating 12 prominent LLMs in the context of COMPL-AI, we reveal shortcomings in existing models and benchmarks, particularly in areas like robustness, safety, diversity, and fairness. This work highlights the need for a shift in focus towards these aspects, encouraging balanced development of LLMs and more comprehensive regulation-aligned benchmarks. Simultaneously, COMPL-AI for the first time demonstrates the possibilities and difficulties of bringing the Act's obligations to a more concrete, technical level. As such, our work can serve as a useful first step towards having actionable recommendations for model providers, and contributes to ongoing efforts of the EU to enable application of the Act, such as the drafting of the GPAI Code of Practice.

Figures (2)

• Figure 1: Overview of COMPL-AI. First, we provide a technical interpretation of the EU AI Act for LLMs, extracting clear technical requirements. Second, we connect these technical requirements to state-of-the-art benchmarks, and collect them in a benchmarking suite. Finally, we use our benchmarking suite to evaluate current LLMs, identifying critical shortcomings in both the models and the current benchmarks from the perspective of the EU AI Act.
• Figure 2: Overview of the structure of the COMPL-AI benchmarking suite. Starting from the six ethical principles of the EU AI Act (left), we extract corresponding technical requirements (middle), and connect those to state-of-the-art LLM benchmarks (right).