Diverse Community Data for Benchmarking Data Privacy Algorithms

TL;DR

The initial set of evaluation results demonstrate the value of these tools for investigations in this field and show the need for a comprehensive open source suite of evaluation metrology for deidentified datasets.

Abstract

The Collaborative Research Cycle (CRC) is a National Institute of Standards and Technology (NIST) benchmarking program intended to strengthen understanding of tabular data deidentification technologies. Deidentification algorithms are vulnerable to the same bias and privacy issues that impact other data analytics and machine learning applications, and can even amplify those issues by contaminating downstream applications. This paper summarizes four CRC contributions: theoretical work on the relationship between diverse populations and challenges for equitable deidentification; public benchmark data focused on diverse populations and challenging features; a comprehensive open source suite of evaluation metrology for deidentified datasets; and an archive of more than 450 deidentified data samples from a broad range of techniques. The initial set of evaluation results demonstrate the value of these tools for investigations in this field.

Figures (9)

• Figure 1: The 24 Features in the Excerpts, and recommended feature subsets.
• Figure 2: The PCA Metric for DP Histogram ($\epsilon = 10$)
• Figure 3: The PCA Metric for PACSynth ($\epsilon = 10$)
• Figure 4: The PCA Metric for PACSynth ($\epsilon = 10$)
• Figure 5: The PCA Metric for MST ($\epsilon = 10$)

Theorems & Definitions (14)

• Definition D.1: Dispersal Ratio
• Lemma D.1
• proof
• Lemma D.2
• proof
• Theorem D.3
• proof
• Theorem D.4
• proof
• Lemma D.5