MultiGraSCCo: A Multilingual Anonymization Benchmark with Annotations of Personal Identifiers

TL;DR

This work creates a multilingual anonymization benchmark in ten languages, using a machine translation methodology that preserves the original annotations and renders names of cities and people in a culturally and contextually appropriate form in each target language.

Abstract

Accessing sensitive patient data for machine learning is challenging due to privacy concerns. Datasets with annotations of personally identifiable information are crucial for developing and testing anonymization systems to enable safe data sharing that complies with privacy regulations. Since accessing real patient data is a bottleneck, synthetic data offers an efficient solution for data scarcity, bypassing privacy regulations that apply to real data. Moreover, neural machine translation can help to create high-quality data for low-resource languages by translating validated real or synthetic data from a high-resource language. In this work, we create a multilingual anonymization benchmark in ten languages, using a machine translation methodology that preserves the original annotations and renders names of cities and people in a culturally and contextually appropriate form in each target language. Our evaluation study with medical professionals confirms the quality of the translations, both in general and with respect to the translation and adaptation of personal information. Our benchmark with over 2,500 annotations of personal information can be used in many applications, including training annotators, validating annotations across institutions without legal complications, and helping improve the performance of automatic personal information detection. We make our benchmark and annotation guidelines available for further research.

Figures (5)

• Figure 1: Our methodology for creating MultiGraSCCo. We begin by annotating IPIs in the dataset to support researchers in building anonymization systems which can be compliant with European privacy regulations. After preprocessing, we use Machine Translation (MT) to translate the texts, together with annotations of direct and indirect identifiers, into 9 languages. Next, we conduct an evaluation study to assess the quality of the translations. Finally, to assess how well privacy-preserving models generalize across languages, we conduct i) monolingual experiments for language-specific performance, ii) cross-lingual experiments to test transfer learning, and iii) multilingual experiments to investigate the benefits of training on the original data in German as well as the translations in one of the target languages.
• Figure 2: Sample translations from German to English, Turkish, Ukrainian, and Arabic with XML tags around the start and end of annotated spans.
• Figure 3: Results of the qualitative evaluation of the automatic translation into the nine languages. Each dimension was evaluated using a Likert scale between 1 and 7.
• Figure 4: The final preprocessing prompt to correct typos and expand abbreviations in the original GraSCCo documents with GPT-4.1.
• Figure 5: The final translation prompt and instructions used to translate the clinical documents with GPT-4.1.