Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GiusBERTo: A Legal Language Model for Personal Data De-identification in Italian Court of Auditors Decisions

Giulio Salierno, Rosamaria Bertè, Luca Attias, Carla Morrone, Dario Pettazzoni, Daniela Battisti

TL;DR

GiusBERTo provides the Italian legal community with an accurate and tailored BERT model for de-identification, balancing privacy and data protection, and achieves 97% token-level accuracy on a held-out test set.

Abstract

Recent advances in Natural Language Processing have demonstrated the effectiveness of pretrained language models like BERT for a variety of downstream tasks. We present GiusBERTo, the first BERT-based model specialized for anonymizing personal data in Italian legal documents. GiusBERTo is trained on a large dataset of Court of Auditors decisions to recognize entities to anonymize, including names, dates, locations, while retaining contextual relevance. We evaluate GiusBERTo on a held-out test set and achieve 97% token-level accuracy. GiusBERTo provides the Italian legal community with an accurate and tailored BERT model for de-identification, balancing privacy and data protection.

GiusBERTo: A Legal Language Model for Personal Data De-identification in Italian Court of Auditors Decisions

TL;DR

GiusBERTo provides the Italian legal community with an accurate and tailored BERT model for de-identification, balancing privacy and data protection, and achieves 97% token-level accuracy on a held-out test set.

Abstract

Recent advances in Natural Language Processing have demonstrated the effectiveness of pretrained language models like BERT for a variety of downstream tasks. We present GiusBERTo, the first BERT-based model specialized for anonymizing personal data in Italian legal documents. GiusBERTo is trained on a large dataset of Court of Auditors decisions to recognize entities to anonymize, including names, dates, locations, while retaining contextual relevance. We evaluate GiusBERTo on a held-out test set and achieve 97% token-level accuracy. GiusBERTo provides the Italian legal community with an accurate and tailored BERT model for de-identification, balancing privacy and data protection.
Paper Structure (14 sections, 13 equations, 4 figures, 6 tables)

This paper contains 14 sections, 13 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Data de-identification in legal decisions
  4. Dataset
  5. Pre-training on masked language task
  6. Documents Matching
  7. Sequences Alignment in Text Documents
  8. Algorithm Steps
  9. Dataset Annotations
  10. Fine-tuning for the de-identification task
  11. Testbed Experimentation
  12. Fine-Tuning Evaluation
  13. Lesson learned & Future work
  14. Conclusions

Figures (4)

  • Figure 1: Data Processing pipeline
  • Figure 2: Epoch vs Loss during training on a Masked Language Modeling (MLM) task. The graph depicts the evolution of training loss over the span of approximately 2.6 epochs. A general declining trend is observed, indicating the optimization of the model parameters over time.
  • Figure 3: Evaluation Metrics Before and After Applying Reweighted Loss Function
  • Figure 4: GiusBERTo Metrics Evaluation