Deep Active Learning with Crowdsourcing Data for Privacy Policy Classification
Wenjun Qiu, David Lie
TL;DR
This work tackles the labeling bottleneck in privacy policy analysis by introducing Calpric, a Crowdsourcing Active Learning framework that segments policies and trains binary classifiers for common data practices. It combines a PPWE-biLSTM classifier with pool-based active learning and crowdsourced labeling on MTurk to efficiently acquire informative labels, enabling high-accuracy privacy data practice classification with only a fraction of the labeling effort. Calpric achieves 97.6% accuracy while reducing labeling effort to about 62% of conventional labeling and automatically mitigates class imbalance by prioritizing informative negative samples. Evaluated on a large corpus of 52K Android app policies, Calpric demonstrates substantial improvements over non-active-learning baselines and surpasses models trained on laboriously labeled data, offering a scalable tool for user-facing privacy policy understanding and regulatory analysis.
Abstract
Privacy policies are statements that notify users of the services' data practices. However, few users are willing to read through policy texts due to the length and complexity. While automated tools based on machine learning exist for privacy policy analysis, to achieve high classification accuracy, classifiers need to be trained on a large labeled dataset. Most existing policy corpora are labeled by skilled human annotators, requiring significant amount of labor hours and effort. In this paper, we leverage active learning and crowdsourcing techniques to develop an automated classification tool named Calpric (Crowdsourcing Active Learning PRIvacy Policy Classifier), which is able to perform annotation equivalent to those done by skilled human annotators with high accuracy while minimizing the labeling cost. Specifically, active learning allows classifiers to proactively select the most informative segments to be labeled. On average, our model is able to achieve the same F1 score using only 62% of the original labeling effort. Calpric's use of active learning also addresses naturally occurring class imbalance in unlabeled privacy policy datasets as there are many more statements stating the collection of private information than stating the absence of collection. By selecting samples from the minority class for labeling, Calpric automatically creates a more balanced training set.