Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Entailment-Driven Privacy Policy Classification with LLMs

Bhanuka Silva, Dishanika Denipitiyage, Suranga Seneviratne, Anirban Mahanti, Aruna Seneviratne

TL;DR

This work tackles the difficulty users face in understanding privacy policies by introducing an entailment-driven framework that classifies policy paragraphs into 12 user-friendly data-practice labels. The system combines an explained classifier, a blank filler, and an entailment verifier to produce both predictions and human-interpretable reasoning, mitigating LLM hallucinations. Evaluated on the OPP-115 dataset, the full pipeline achieves a macro-F1 of approximately 0.63 and outperforms several language-generation baselines by notable margins, while providing substantially improved explainability (56–58% overlap with legal annotations) compared to embedding-based methods. The approach demonstrates a practical path toward more transparent and user-friendly privacy policy tools, with potential extensions to broader datasets and domain adaptation.

Abstract

While many online services provide privacy policies for end users to read and understand what personal data are being collected, these documents are often lengthy and complicated. As a result, the vast majority of users do not read them at all, leading to data collection under uninformed consent. Several attempts have been made to make privacy policies more user friendly by summarising them, providing automatic annotations or labels for key sections, or by offering chat interfaces to ask specific questions. With recent advances in Large Language Models (LLMs), there is an opportunity to develop more effective tools to parse privacy policies and help users make informed decisions. In this paper, we propose an entailment-driven LLM based framework to classify paragraphs of privacy policies into meaningful labels that are easily understood by users. The results demonstrate that our framework outperforms traditional LLM methods, improving the F1 score in average by 11.2%. Additionally, our framework provides inherently explainable and meaningful predictions.

Entailment-Driven Privacy Policy Classification with LLMs

TL;DR

This work tackles the difficulty users face in understanding privacy policies by introducing an entailment-driven framework that classifies policy paragraphs into 12 user-friendly data-practice labels. The system combines an explained classifier, a blank filler, and an entailment verifier to produce both predictions and human-interpretable reasoning, mitigating LLM hallucinations. Evaluated on the OPP-115 dataset, the full pipeline achieves a macro-F1 of approximately 0.63 and outperforms several language-generation baselines by notable margins, while providing substantially improved explainability (56–58% overlap with legal annotations) compared to embedding-based methods. The approach demonstrates a practical path toward more transparent and user-friendly privacy policy tools, with potential extensions to broader datasets and domain adaptation.

Abstract

While many online services provide privacy policies for end users to read and understand what personal data are being collected, these documents are often lengthy and complicated. As a result, the vast majority of users do not read them at all, leading to data collection under uninformed consent. Several attempts have been made to make privacy policies more user friendly by summarising them, providing automatic annotations or labels for key sections, or by offering chat interfaces to ask specific questions. With recent advances in Large Language Models (LLMs), there is an opportunity to develop more effective tools to parse privacy policies and help users make informed decisions. In this paper, we propose an entailment-driven LLM based framework to classify paragraphs of privacy policies into meaningful labels that are easily understood by users. The results demonstrate that our framework outperforms traditional LLM methods, improving the F1 score in average by 11.2%. Additionally, our framework provides inherently explainable and meaningful predictions.
Paper Structure (24 sections, 4 figures, 3 tables)

This paper contains 24 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Empirical Studies on Privacy Policies
  4. Analysing Privacy Policies Using NLP Techniques
  5. Large Language Models (LLMs)
  6. Our Framework
  7. Explained Classifier
  8. Blank Filler
  9. Entailment Verifier
  10. Experimental Setup
  11. Modules of the Framework
  12. OPP-115 Dataset
  13. Training/Testing Pipeline
  14. Baselines
  15. Embedding-Based Classification Models
  16. ...and 9 more sections

Figures (4)

  • Figure 1: Four stages of the entailment-driven privacy policy classification. Depicted paragraph is from our test-dataset and all of the outputs generated at stage 1 would undergo stage 2 to 4 separately.
  • Figure 2: End to end pipeline of our method. Shaded in green are decoder models and shaded in blue is an encoder model. Phase 1 represents the training of explained classifier and blank filler. Phase 2 represents the training of the entailment verifier, for which we use the already trained modules from phase 1 in inference mode. Phase 3 represents all three modules working in inference mode with test dataset.
  • Figure 3: Two example paragraphs from train-set depicting the annotated segment and the relevant, data-practice label*, data-attribute label**, and data-value label***
  • Figure 4: Explainability visualised: (a): our method's generated, (b): randomly sampled, (c): LIME based PrivBERT interpretations, compared with legal expert annotations. *Subfigure (c) is not colour mapped. **All sub-figures: outlined in red colour are the samples that have less than 10% overlap with legal expert annotations. Outlined in black are the samples having more than 50% overlap.