Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating the Usability of Differential Privacy Tools with Data Practitioners

Ivoline C. Ngong, Brad Stenger, Joseph P. Near, Yuanyuan Feng

TL;DR

The paper tackles the usability barrier to real-world differential privacy by conducting the first cross-tool usability study of four Python-based DP tools with 24 data practitioners. It evaluates DP understanding, implementation, and user satisfaction using learnability, efficiency, error prevention, and SUS/NPS metrics, revealing that novices can gain DP comprehension through hands-on tasks while API design and documentation critically shape success. DiffPrivLib typically yields higher task completion but can permit DP violations due to flexible defaults, whereas OpenDP shows stronger DP-violation prevention at the cost of usability; overall, tool design and educational resources determine adoption potential. The authors provide evidence-based recommendations—improved navigation, DP-specific examples, clearer error messages, intuitive APIs, and DP foundations education—to broaden DP adoption in industry and practice.

Abstract

Differential privacy (DP) has become the gold standard in privacy-preserving data analytics, but implementing it in real-world datasets and systems remains challenging. Recently developed DP tools aim to make DP implementation easier, but limited research has investigated these DP tools' usability. Through a usability study with 24 US data practitioners with varying prior DP knowledge, we evaluated the usability of four Python-based open-source DP tools: DiffPrivLib, Tumult Analytics, PipelineDP, and OpenDP. Our results suggest that using DP tools in this study may help DP novices better understand DP; that Application Programming Interface (API) design and documentation are vital for successful DP implementation; and that user satisfaction correlates with how well participants completed study tasks with these DP tools. We provide evidence-based recommendations to improve DP tools' usability to broaden DP adoption.

Evaluating the Usability of Differential Privacy Tools with Data Practitioners

TL;DR

The paper tackles the usability barrier to real-world differential privacy by conducting the first cross-tool usability study of four Python-based DP tools with 24 data practitioners. It evaluates DP understanding, implementation, and user satisfaction using learnability, efficiency, error prevention, and SUS/NPS metrics, revealing that novices can gain DP comprehension through hands-on tasks while API design and documentation critically shape success. DiffPrivLib typically yields higher task completion but can permit DP violations due to flexible defaults, whereas OpenDP shows stronger DP-violation prevention at the cost of usability; overall, tool design and educational resources determine adoption potential. The authors provide evidence-based recommendations—improved navigation, DP-specific examples, clearer error messages, intuitive APIs, and DP foundations education—to broaden DP adoption in industry and practice.

Abstract

Differential privacy (DP) has become the gold standard in privacy-preserving data analytics, but implementing it in real-world datasets and systems remains challenging. Recently developed DP tools aim to make DP implementation easier, but limited research has investigated these DP tools' usability. Through a usability study with 24 US data practitioners with varying prior DP knowledge, we evaluated the usability of four Python-based open-source DP tools: DiffPrivLib, Tumult Analytics, PipelineDP, and OpenDP. Our results suggest that using DP tools in this study may help DP novices better understand DP; that Application Programming Interface (API) design and documentation are vital for successful DP implementation; and that user satisfaction correlates with how well participants completed study tasks with these DP tools. We provide evidence-based recommendations to improve DP tools' usability to broaden DP adoption.
Paper Structure (67 sections, 1 equation, 7 figures, 5 tables)

This paper contains 67 sections, 1 equation, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. DP and Implementation Challenges.
  4. Existing DP Tools.
  5. User Research around DP Understanding.
  6. Usability around DP Implementation.
  7. Usability of Non-DP Tools.
  8. Methods and Study Design
  9. Selection of Differential Privacy Tools
  10. Study Procedures
  11. Recruitment & Screening
  12. Pre-task Procedures
  13. Usability Testing Tasks
  14. Post-task Procedures
  15. Usability Measurements and Data Analysis
  16. ...and 52 more sections

Figures (7)

  • Figure 1: Total number of correct answers to DP knowledge questions before and after study tasks.
  • Figure 2: Useful sources that support participants' DP understanding, by tool (a) and by expertise level (b).
  • Figure 3: Learnability of DP tools measured by (a) task completion rates and (b) task correctness rates. Each cell represents the percentage of participants who completed or correctly completed the task using the tool.
  • Figure 4: Average task completion time: (a) by tool (b) by expertise level.
  • Figure 5: Factors helping task completion by tool and expertise.
  • ...and 2 more figures