Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data Annotation Quality Problems in AI-Enabled Perception System Development

Hina Saeeda, Tommy Johansson, Mazen Mohamad, Eric Knauss

TL;DR

Data annotation quality is a critical yet underexplored bottleneck in AI-enabled perception systems for autonomous driving. The authors conduct a multi-organisational European case study using 19 interviews to develop an empirically grounded taxonomy of $18$ annotation errors across three data-quality dimensions: Completeness, Accuracy, and Consistency, and validate its industrial usefulness as a diagnostic tool. By linking upstream annotation faults to downstream perception performance, the work reframes annotation quality as a lifecycle and supply-chain issue suitable for SE4AI governance, onboarding, and continuous improvement across OEMs and suppliers. These contributions provide a shared vocabulary, a practical failure-mode catalogue, and guidance for integrating data quality into lifecycle engineering, audits, and risk-based prioritisation in safety-critical perception systems. Future work points to automated QA, cross-geography validation, and quantitative links between error types and model outcomes to strengthen lifecycle assurance.

Abstract

Data annotation is essential but highly error-prone in the development of AI-enabled perception systems (AIePS) for automated driving, and its quality directly influences model performance, safety, and reliability. However, the industry lacks empirical insights into how annotation errors emerge and spread across the multi-organisational automotive supply chain. This study addresses this gap through a multi-organisation case study involving six companies and four research institutes across Europe and the UK. Based on 19 semi-structured interviews with 20 experts (50 hours of transcripts) and a six-phase thematic analysis, we develop a taxonomy of 18 recurring annotation error types across three data-quality dimensions: completeness (e.g., attribute omission, missing feedback loops, edge-case omissions, selection bias), accuracy (e.g., mislabelling, bounding-box inaccuracies, granularity mismatches, bias-driven errors), and consistency (e.g., inter-annotator disagreement, ambiguous instructions, misaligned hand-offs, cross-modality inconsistencies). The taxonomy was validated with industry practitioners, who reported its usefulness for root-cause analysis, supplier quality reviews, onboarding, and improving annotation guidelines. They described it as a failure-mode catalogue similar to FMEA. By conceptualising annotation quality as a lifecycle and supply-chain issue, this study contributes to SE4AI by offering a shared vocabulary, diagnostic toolset, and actionable guidance for building trustworthy AI-enabled perception systems.

Data Annotation Quality Problems in AI-Enabled Perception System Development

TL;DR

Data annotation quality is a critical yet underexplored bottleneck in AI-enabled perception systems for autonomous driving. The authors conduct a multi-organisational European case study using 19 interviews to develop an empirically grounded taxonomy of annotation errors across three data-quality dimensions: Completeness, Accuracy, and Consistency, and validate its industrial usefulness as a diagnostic tool. By linking upstream annotation faults to downstream perception performance, the work reframes annotation quality as a lifecycle and supply-chain issue suitable for SE4AI governance, onboarding, and continuous improvement across OEMs and suppliers. These contributions provide a shared vocabulary, a practical failure-mode catalogue, and guidance for integrating data quality into lifecycle engineering, audits, and risk-based prioritisation in safety-critical perception systems. Future work points to automated QA, cross-geography validation, and quantitative links between error types and model outcomes to strengthen lifecycle assurance.

Abstract

Data annotation is essential but highly error-prone in the development of AI-enabled perception systems (AIePS) for automated driving, and its quality directly influences model performance, safety, and reliability. However, the industry lacks empirical insights into how annotation errors emerge and spread across the multi-organisational automotive supply chain. This study addresses this gap through a multi-organisation case study involving six companies and four research institutes across Europe and the UK. Based on 19 semi-structured interviews with 20 experts (50 hours of transcripts) and a six-phase thematic analysis, we develop a taxonomy of 18 recurring annotation error types across three data-quality dimensions: completeness (e.g., attribute omission, missing feedback loops, edge-case omissions, selection bias), accuracy (e.g., mislabelling, bounding-box inaccuracies, granularity mismatches, bias-driven errors), and consistency (e.g., inter-annotator disagreement, ambiguous instructions, misaligned hand-offs, cross-modality inconsistencies). The taxonomy was validated with industry practitioners, who reported its usefulness for root-cause analysis, supplier quality reviews, onboarding, and improving annotation guidelines. They described it as a failure-mode catalogue similar to FMEA. By conceptualising annotation quality as a lifecycle and supply-chain issue, this study contributes to SE4AI by offering a shared vocabulary, diagnostic toolset, and actionable guidance for building trustworthy AI-enabled perception systems.

Paper Structure

This paper contains 8 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Research Methodology
  4. Findings
  5. RQ1: Data Annotation Errors, their Types, Causes and Impacts in AIePS Development
  6. RQ2: Experts' Validation of the Data Annotation Errors Taxonomy
  7. Threats to Validity
  8. Conclusions and Discussion

Figures (5)

  • Figure 1: Research method followed.
  • Figure 2: Taxonomy of data annotation errors in AI-enabled perception systems, organised into three main categories: Completeness, Accuracy, and Consistency.
  • Figure 3: High mentioned completeness errors.
  • Figure 4: High mentioned accuracy errors.
  • Figure 5: High mentioned consistency errors.