Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scalable multilingual PII annotation for responsible AI in LLMs

Bharti Meena, Joanna Skubisz, Harshit Rajgarhia, Nand Dave, Kiran Ganesh, Shivali Dalmia, Abhishek Mukherji, Vasudevan Sundarababu

TL;DR

This work tackles the challenge of reliably annotating PII across 13 underrepresented locales to support responsible AI in LLMs. It introduces a phased, human-in-the-loop data curation framework that covers ~336 locale-specific PII types and scales to over 40,000 text samples, with a pilot-training-production pipeline and a semi-automatic augmentation approach for HiTL prompts. By measuring inter-annotator agreement and conducting root-cause analyses, the authors achieve notable improvements in $Recall$ and reductions in $FPR$ across locales, yielding high-fidelity datasets for supervised fine-tuning. The framework's analytics-driven design enhances annotation quality and downstream robustness, supporting privacy-preserving LLM deployment under diverse regulatory contexts.

Abstract

As Large Language Models (LLMs) gain wider adoption, ensuring their reliable handling of Personally Identifiable Information (PII) across diverse regulatory contexts has become essential. This work introduces a scalable multilingual data curation framework designed for high-quality PII annotation across 13 underrepresented locales, covering approximately 336 locale-specific PII types. Our phased, human-in-the-loop annotation methodology combines linguistic expertise with rigorous quality assurance, leading to substantial improvements in recall and false positive rates from pilot, training, and production phases. By leveraging inter-annotator agreement metrics and root-cause analysis, the framework systematically uncovers and resolves annotation inconsistencies, resulting in high-fidelity datasets suitable for supervised LLM fine-tuning. Beyond reporting empirical gains, we highlight common annotator challenges in multilingual PII labeling and demonstrate how iterative, analytics-driven pipelines can enhance both annotation quality and downstream model reliability.

Scalable multilingual PII annotation for responsible AI in LLMs

TL;DR

This work tackles the challenge of reliably annotating PII across 13 underrepresented locales to support responsible AI in LLMs. It introduces a phased, human-in-the-loop data curation framework that covers ~336 locale-specific PII types and scales to over 40,000 text samples, with a pilot-training-production pipeline and a semi-automatic augmentation approach for HiTL prompts. By measuring inter-annotator agreement and conducting root-cause analyses, the authors achieve notable improvements in and reductions in across locales, yielding high-fidelity datasets for supervised fine-tuning. The framework's analytics-driven design enhances annotation quality and downstream robustness, supporting privacy-preserving LLM deployment under diverse regulatory contexts.

Abstract

As Large Language Models (LLMs) gain wider adoption, ensuring their reliable handling of Personally Identifiable Information (PII) across diverse regulatory contexts has become essential. This work introduces a scalable multilingual data curation framework designed for high-quality PII annotation across 13 underrepresented locales, covering approximately 336 locale-specific PII types. Our phased, human-in-the-loop annotation methodology combines linguistic expertise with rigorous quality assurance, leading to substantial improvements in recall and false positive rates from pilot, training, and production phases. By leveraging inter-annotator agreement metrics and root-cause analysis, the framework systematically uncovers and resolves annotation inconsistencies, resulting in high-fidelity datasets suitable for supervised LLM fine-tuning. Beyond reporting empirical gains, we highlight common annotator challenges in multilingual PII labeling and demonstrate how iterative, analytics-driven pipelines can enhance both annotation quality and downstream model reliability.

Paper Structure

This paper contains 21 sections, 2 equations, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. PII Data Protection
  3. Challenges in Data Annotation for LLM SFT
  4. Key Contributions
  5. Related Work
  6. PII Annotation in Multilingual and Low-Resource Settings
  7. Named Entity Recognition for PII Extraction
  8. Human-in-the-Loop Annotation Frameworks
  9. Annotator Agreement and Data Quality for LLM Fine-Tuning
  10. Background
  11. PII definition and locale-specific variations
  12. Phased Execution of PII Annotation
  13. PII Annotation Process
  14. Methodological Approach
  15. Agreement Definition
  16. ...and 6 more sections

Figures (6)

  • Figure 1: Annotator workflow. (Left) User Interface of labeled prompt.(Right)
  • Figure 2: Descriptive statistics of error categories in PowerBI identified in the QA review per locale. (Left) Quality score of Annotators.(Right)
  • Figure 3: Annotation Matrix for Pilot Phase for pl-PL locale. (Left); Production Phase for pl-PL locale. (Right)
  • Figure 4: Normalized domain distribution across locales. (Left) Normalized prompt length distribution across locales.(Right)
  • Figure 5: Normalized distribution of PII types across 13 locales.
  • ...and 1 more figures