Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond the Checkbox: Strengthening DSA Compliance Through Social Media Algorithmic Auditing

Sara Solarova, Matúš Mesarčík, Branislav Pecher, Ivan Srba

TL;DR

This paper interrogates the first wave of EU DSA audits of major social platforms, revealing substantial methodological inconsistencies and shallow technical depth in evaluating AI-driven systems. It argues that traditional, point-in-time audits are ill-suited to dynamic recommender and advertising ecosystems and proposes algorithmic auditing—behavioural, simulation-based testing—as a robust complement. Through qualitative analysis of four public reports, the study highlights gaps in measuring meaningful user control, minors protection, and sensitive-data advertising, and demonstrates how coordinated, long-term, scenario-driven audits could improve accountability. The work underlines the practical need for standardized, transparent, and external auditing practices to effectively enforce DSA obligations while acknowledging challenges in realism, replicability, and integration with regulatory processes.

Abstract

Algorithms of online platforms are required under the Digital Services Act (DSA) to comply with specific obligations concerning algorithmic transparency, user protection and privacy. To verify compliance with these requirements, DSA mandates platforms to undergo independent audits. Little is known about current auditing practices and their effectiveness in ensuring such compliance. To this end, we bridge regulatory and technical perspectives by critically examining selected audit reports across three critical algorithmic-related provisions: restrictions on profiling minors, transparency in recommender systems, and limitations on targeted advertising using sensitive data. Our analysis shows significant inconsistencies in methodologies and lack of technical depth when evaluating AI-powered systems. To enhance the depth, scale, and independence of compliance assessments, we propose to employ algorithmic auditing -- a process of behavioural assessment of AI algorithms by means of simulating user behaviour, observing algorithm responses and analysing them for audited phenomena.

Beyond the Checkbox: Strengthening DSA Compliance Through Social Media Algorithmic Auditing

TL;DR

This paper interrogates the first wave of EU DSA audits of major social platforms, revealing substantial methodological inconsistencies and shallow technical depth in evaluating AI-driven systems. It argues that traditional, point-in-time audits are ill-suited to dynamic recommender and advertising ecosystems and proposes algorithmic auditing—behavioural, simulation-based testing—as a robust complement. Through qualitative analysis of four public reports, the study highlights gaps in measuring meaningful user control, minors protection, and sensitive-data advertising, and demonstrates how coordinated, long-term, scenario-driven audits could improve accountability. The work underlines the practical need for standardized, transparent, and external auditing practices to effectively enforce DSA obligations while acknowledging challenges in realism, replicability, and integration with regulatory processes.

Abstract

Algorithms of online platforms are required under the Digital Services Act (DSA) to comply with specific obligations concerning algorithmic transparency, user protection and privacy. To verify compliance with these requirements, DSA mandates platforms to undergo independent audits. Little is known about current auditing practices and their effectiveness in ensuring such compliance. To this end, we bridge regulatory and technical perspectives by critically examining selected audit reports across three critical algorithmic-related provisions: restrictions on profiling minors, transparency in recommender systems, and limitations on targeted advertising using sensitive data. Our analysis shows significant inconsistencies in methodologies and lack of technical depth when evaluating AI-powered systems. To enhance the depth, scale, and independence of compliance assessments, we propose to employ algorithmic auditing -- a process of behavioural assessment of AI algorithms by means of simulating user behaviour, observing algorithm responses and analysing them for audited phenomena.
Paper Structure (17 sections, 2 figures, 1 table)

This paper contains 17 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. DSA Obligations and Audits Verifying Regulation Compliance
  4. Existing Audit Methodologies
  5. Methodology
  6. Document Corpus and Selection Criteria
  7. Analytical Framework and Coding Procedure
  8. Researcher Positionality and Limitations
  9. Analysis of Audit Reports
  10. Case 1 -- Recommender System Transparency and Non-Profiling Options
  11. Case 2 -- Protection of Minors
  12. Case 3 -- Advertisements
  13. Identification of Challenges in Auditing AI-based Systems
  14. Social Media Algorithmic Auditing
  15. Relevance to Identified Challenges in Auditing AI-based Systems
  16. ...and 2 more sections

Figures (2)

  • Figure 1: A DSA tiered system of due diligence obligations (inspired by HusovecLaguna2022DigitalServicesAct). It is structured as a regulatory pyramid: 1) at its base lie the universal and basic obligations that apply to all intermediary services, complemented by advanced obligations for hosting providers and online platforms; 2) at the top of this hierarchy stand the special obligations tailored to VLOPs and VLOSEs, which are specifically crafted to address systemic risks arising from their scale and societal reach.
  • Figure 2: A process of typical algorithmic auditing approach. It consists of the following steps: 1) by proceeding from an audit question, audit scenarios consisting of user profiles and user actions are created; 2) during audit execution and evaluation, bots/human agents execute such audit scenarios, platform responses are recorded, and further investigated for the presence of audited phenomena; 3) audit reports are produced.