Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Calpric: Inclusive and Fine-grain Labeling of Privacy Policies with Crowdsourcing and Active Learning

Wenjun Qiu, David Lie, Lisa Austin

TL;DR

Calpric addresses the challenge of obtaining large, balanced, fine-grained training data for privacy-policy analysis by integrating automatic text segmentation, crowdsourcing, and active learning. The approach yields the Calpric Privacy Policy Corpus (CPPS) with 16,856 labeled segments across 9 data categories and 3 data actions, at a low labeling cost (approximately $0.92-$1.71 per label). It introduces PriBERT-based Segmenters and 27 Action Models, enabling precise differentiation of denials, opt-outs, and ambiguities, and demonstrates strong end-to-end performance (average F1 ≈ 0.86, accuracy ≈ 0.90) and improved minority-category coverage. The work shows substantial labeling savings, improved class balance, and competitive costs against trained annotators, with practical implications for scalable privacy-policy analysis and compliance tooling, including Android app policies.

Abstract

A significant challenge to training accurate deep learning models on privacy policies is the cost and difficulty of obtaining a large and comprehensive set of training data. To address these challenges, we present Calpric , which combines automatic text selection and segmentation, active learning and the use of crowdsourced annotators to generate a large, balanced training set for privacy policies at low cost. Automated text selection and segmentation simplifies the labeling task, enabling untrained annotators from crowdsourcing platforms, like Amazon's Mechanical Turk, to be competitive with trained annotators, such as law students, and also reduces inter-annotator agreement, which decreases labeling cost. Having reliable labels for training enables the use of active learning, which uses fewer training samples to efficiently cover the input space, further reducing cost and improving class and data category balance in the data set. The combination of these techniques allows Calpric to produce models that are accurate over a wider range of data categories, and provide more detailed, fine-grain labels than previous work. Our crowdsourcing process enables Calpric to attain reliable labeled data at a cost of roughly $0.92-$1.71 per labeled text segment. Calpric 's training process also generates a labeled data set of 16K privacy policy text segments across 9 Data categories with balanced positive and negative samples.

Calpric: Inclusive and Fine-grain Labeling of Privacy Policies with Crowdsourcing and Active Learning

TL;DR

Calpric addresses the challenge of obtaining large, balanced, fine-grained training data for privacy-policy analysis by integrating automatic text segmentation, crowdsourcing, and active learning. The approach yields the Calpric Privacy Policy Corpus (CPPS) with 16,856 labeled segments across 9 data categories and 3 data actions, at a low labeling cost (approximately 1.71 per label). It introduces PriBERT-based Segmenters and 27 Action Models, enabling precise differentiation of denials, opt-outs, and ambiguities, and demonstrates strong end-to-end performance (average F1 ≈ 0.86, accuracy ≈ 0.90) and improved minority-category coverage. The work shows substantial labeling savings, improved class balance, and competitive costs against trained annotators, with practical implications for scalable privacy-policy analysis and compliance tooling, including Android app policies.

Abstract

A significant challenge to training accurate deep learning models on privacy policies is the cost and difficulty of obtaining a large and comprehensive set of training data. To address these challenges, we present Calpric , which combines automatic text selection and segmentation, active learning and the use of crowdsourced annotators to generate a large, balanced training set for privacy policies at low cost. Automated text selection and segmentation simplifies the labeling task, enabling untrained annotators from crowdsourcing platforms, like Amazon's Mechanical Turk, to be competitive with trained annotators, such as law students, and also reduces inter-annotator agreement, which decreases labeling cost. Having reliable labels for training enables the use of active learning, which uses fewer training samples to efficiently cover the input space, further reducing cost and improving class and data category balance in the data set. The combination of these techniques allows Calpric to produce models that are accurate over a wider range of data categories, and provide more detailed, fine-grain labels than previous work. Our crowdsourcing process enables Calpric to attain reliable labeled data at a cost of roughly 1.71 per labeled text segment. Calpric 's training process also generates a labeled data set of 16K privacy policy text segments across 9 Data categories with balanced positive and negative samples.
Paper Structure (39 sections, 1 equation, 6 figures, 12 tables)

This paper contains 39 sections, 1 equation, 6 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Challenges
  3. Inclusive and Fine-grained Classification
  4. Cost and Imbalance in Training Data
  5. Class Imbalance
  6. Labeling Cost
  7. Segmentation
  8. Design Overview
  9. The Calpric Pipeline
  10. Data Preparation
  11. Segmenter
  12. Category Models
  13. Contextualizer
  14. Action Models
  15. Active Learning and Crowdsourcing
  16. ...and 24 more sections

Figures (6)

  • Figure 1: A fully labeled privacy policy segment with data category (contact information), data action (third party sharing), and its corresponding action mode (does not share).
  • Figure 2: Privacy Policy segments with multiple labels
  • Figure 3: A simplified overview of the active learning system
  • Figure 4: The action classification process to produce a label on the given contact sample segment
  • Figure 5: (a) an example result of Accepted labels vs. Labeling Requests at different Acceptance Thresholds (AT) (data category: contact); (b) average Acceptance Rate (AR) and F1 performance for different ATs
  • ...and 1 more figures