Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Empirical Study on Compliance with Ranking Transparency in the Software Documentation of EU Online Platforms

Francesco Sovrano, Michaël Lognoul, Alberto Bacchelli

TL;DR

This study evaluates ranking transparency documentation under the EU P2B Regulation across six major platforms, combining manual expert/layperson assessments with automated tools. It introduces and tests two automated approaches—a ChatGPT-based assessor and a DoX-based answer-retrieval method—against human judgments, showing moderate alignment and highlighting variability in platform compliance. DoX demonstrates particular strength with lengthy documents, while ChatGPT performance depends on model and prompting; the authors advocate a hybrid workflow that leverages automated tools to flag non-compliance and guide expert review. The work contributes a replication package, a regulation-grounded checklist, and a DoX-based framework that together advance scalable monitoring of platform transparency and support regulatory efforts toward more equitable digital ecosystems (SDG 10.3).

Abstract

Compliance with the European Union's Platform-to-Business (P2B) Regulation is challenging for online platforms, and assessing their compliance can be difficult for public authorities. This is partly due to the lack of automated tools for assessing the information (e.g., software documentation) platforms provide concerning ranking transparency. Our study tackles this issue in two ways. First, we empirically evaluate the compliance of six major platforms (Amazon, Bing, Booking, Google, Tripadvisor, and Yahoo), revealing substantial differences in their documentation. Second, we introduce and test automated compliance assessment tools based on ChatGPT and information retrieval technology. These tools are evaluated against human judgments, showing promising results as reliable proxies for compliance assessments. Our findings could help enhance regulatory compliance and align with the United Nations Sustainable Development Goal 10.3, which seeks to reduce inequality, including business disparities, on these platforms.

An Empirical Study on Compliance with Ranking Transparency in the Software Documentation of EU Online Platforms

TL;DR

This study evaluates ranking transparency documentation under the EU P2B Regulation across six major platforms, combining manual expert/layperson assessments with automated tools. It introduces and tests two automated approaches—a ChatGPT-based assessor and a DoX-based answer-retrieval method—against human judgments, showing moderate alignment and highlighting variability in platform compliance. DoX demonstrates particular strength with lengthy documents, while ChatGPT performance depends on model and prompting; the authors advocate a hybrid workflow that leverages automated tools to flag non-compliance and guide expert review. The work contributes a replication package, a regulation-grounded checklist, and a DoX-based framework that together advance scalable monitoring of platform transparency and support regulatory efforts toward more equitable digital ecosystems (SDG 10.3).

Abstract

Compliance with the European Union's Platform-to-Business (P2B) Regulation is challenging for online platforms, and assessing their compliance can be difficult for public authorities. This is partly due to the lack of automated tools for assessing the information (e.g., software documentation) platforms provide concerning ranking transparency. Our study tackles this issue in two ways. First, we empirically evaluate the compliance of six major platforms (Amazon, Bing, Booking, Google, Tripadvisor, and Yahoo), revealing substantial differences in their documentation. Second, we introduce and test automated compliance assessment tools based on ChatGPT and information retrieval technology. These tools are evaluated against human judgments, showing promising results as reliable proxies for compliance assessments. Our findings could help enhance regulatory compliance and align with the United Nations Sustainable Development Goal 10.3, which seeks to reduce inequality, including business disparities, on these platforms.
Paper Structure (18 sections, 1 equation, 2 figures, 4 tables)

This paper contains 18 sections, 1 equation, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. P2B Regulation & Guidelines
  5. How to Measure Explainability
  6. Study & Checklist Design
  7. The Study
  8. The Checklist
  9. Automated Assessment
  10. ChatGPT-based Assessment
  11. Assessing Explanation Quality with DoX
  12. Results
  13. Comprehensive Review by Legal Experts
  14. Large-scale Manual Assessment
  15. Automated Assessments
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Average agreement rates for participants reviewing documentation on Tripadvisor and Booking, considering review durations from five to zero minutes. Includes statistically significant p-values for agreement rate comparisons between platforms.
  • Figure 2: Boxplot comparing 'Explanatory Relevance', 'Pertinence', and 'DoX' by experts' answers. Annotations indicate $p$-values and rank biserial correlation ($r$).