Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PoliGraph: Automated Privacy Policy Analysis using Knowledge Graphs (Journal Version)

Hao Cui, Rahmadi Trimananda, Scott Jordan, Athina Markopoulou

TL;DR

PoliGraph reframes privacy policies as a holistic knowledge graph that connects data types, entities, and purposes across the entire text, enabling cross-sentence inferences beyond prior sentence-level analyses. PoliGrapher uses structured linguistic analysis to extract PoliGraphs and demonstrates substantial recall gains with high precision, outperforming prior policy analyzers on collection statements. The framework introduces local versus global ontologies to ground term definitions and enable corpus-level summarization, term-definition validation, and data flow consistency checks. The authors further extend the approach with PoliGrapher-LM, an LLM-based variant that improves recall via prompting and reflection while highlighting cost and coverage trade-offs. Overall, PoliGraph enables new policy analyses, including contradiction detection and data-flow-to-policy consistency, and points to future directions in end-to-end LLM workflows and machine-readable policy representations.

Abstract

Privacy policies disclose how an organization collects and handles personal information. Recent work has made progress in leveraging natural language processing (NLP) to automate privacy policy analysis and extract data collection statements from different sentences, considered in isolation from each other. In this paper, we view and analyze, for the first time, the entire text of a privacy policy in an integrated way. In terms of methodology: (1) we define PoliGraph, a type of knowledge graph that captures statements in a policy as relations between different parts of the text; and (2) we revisit the notion of ontologies, previously defined in heuristic ways, to capture subsumption relations between terms. We make a clear distinction between local and global ontologies to capture the context of individual policies, application domains, and privacy laws. We develop PoliGrapher, an NLP tool to automatically extract PoliGraph from the text using linguistic analysis. Using a public dataset for evaluation, we show that PoliGrapher identifies 40% more collection statements than prior state-of-the-art, with 97% precision. In terms of applications, PoliGraph enables automated analysis of a corpus of policies and allows us to: (1) reveal common patterns in the texts across different policies, and (2) assess the correctness of the terms as defined within a policy. We also apply PoliGraph to: (3) detect contradictions in a policy, where we show false alarms by prior work, and (4) analyze the consistency of policies and network traffic, where we identify significantly more clear disclosures than prior work. Finally, leveraging the capabilities of the emerging large language models (LLMs), we also present PoliGrapher-LM, a tool that uses LLM prompting instead of NLP linguistic analysis, to extract PoliGraph from the policy text, and we show that it further improves coverage.

PoliGraph: Automated Privacy Policy Analysis using Knowledge Graphs (Journal Version)

TL;DR

PoliGraph reframes privacy policies as a holistic knowledge graph that connects data types, entities, and purposes across the entire text, enabling cross-sentence inferences beyond prior sentence-level analyses. PoliGrapher uses structured linguistic analysis to extract PoliGraphs and demonstrates substantial recall gains with high precision, outperforming prior policy analyzers on collection statements. The framework introduces local versus global ontologies to ground term definitions and enable corpus-level summarization, term-definition validation, and data flow consistency checks. The authors further extend the approach with PoliGrapher-LM, an LLM-based variant that improves recall via prompting and reflection while highlighting cost and coverage trade-offs. Overall, PoliGraph enables new policy analyses, including contradiction detection and data-flow-to-policy consistency, and points to future directions in end-to-end LLM workflows and machine-readable policy representations.

Abstract

Privacy policies disclose how an organization collects and handles personal information. Recent work has made progress in leveraging natural language processing (NLP) to automate privacy policy analysis and extract data collection statements from different sentences, considered in isolation from each other. In this paper, we view and analyze, for the first time, the entire text of a privacy policy in an integrated way. In terms of methodology: (1) we define PoliGraph, a type of knowledge graph that captures statements in a policy as relations between different parts of the text; and (2) we revisit the notion of ontologies, previously defined in heuristic ways, to capture subsumption relations between terms. We make a clear distinction between local and global ontologies to capture the context of individual policies, application domains, and privacy laws. We develop PoliGrapher, an NLP tool to automatically extract PoliGraph from the text using linguistic analysis. Using a public dataset for evaluation, we show that PoliGrapher identifies 40% more collection statements than prior state-of-the-art, with 97% precision. In terms of applications, PoliGraph enables automated analysis of a corpus of policies and allows us to: (1) reveal common patterns in the texts across different policies, and (2) assess the correctness of the terms as defined within a policy. We also apply PoliGraph to: (3) detect contradictions in a policy, where we show false alarms by prior work, and (4) analyze the consistency of policies and network traffic, where we identify significantly more clear disclosures than prior work. Finally, leveraging the capabilities of the emerging large language models (LLMs), we also present PoliGrapher-LM, a tool that uses LLM prompting instead of NLP linguistic analysis, to extract PoliGraph from the policy text, and we show that it further improves coverage.
Paper Structure (127 sections, 11 figures, 11 tables)

This paper contains 127 sections, 11 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Privacy Policies
  3. NLP Analysis and Limitations
  4. The PoliGraph Framework
  5. PoliGrapher: Generating PoliGraph using Linguistic Analysis
  6. Applications
  7. PoliGrapher-LM: Generating PoliGraph using LLMs
  8. Overview
  9. Related Work
  10. Formalizing Policies
  11. Policy Analysis
  12. Automated Policy Analysis
  13. Knowledge Graphs
  14. The PoliGraph Framework
  15. Defining PoliGraph
  16. ...and 112 more sections

Figures (11)

  • Figure 1: Example of a privacy policy and analysis approaches.(a) The excerpt is from the policy of KAYAK kayak-policy. It contains sections and lists, regarding: what is collected (data type), how it is used (purpose), who receives the information (entity), and references across sentences (e.g., "personal information" relates to other data types; "this information" refers to "location"). (b) Prior work extracts elements found in each sentence, mainly data types and entities, as disconnected tuples. Purposes can also be extracted to extend the tuple bui2021consistencytrimananda2022ovrseen. (c) PoliGraph is a knowledge graph that encodes data types, entities, and purposes; and two types of relations between them (collection and subsumption), possibly specified across different sentences. A [1.0]COLLECT edge represents that a data type is collected by an entity, while edge attributes represent the purposes of that collection. [1.0]SUBSUME edges represent the subsumption relations between generic and specific terms.
  • Figure 2: Global Data Ontology based on the CCPA.
  • Figure 3: Global Entity Ontology based on tracker-radarcrunchbase-data.
  • Figure 4: Overview of PoliGrapher implementation. First, PoliGrapher preprocesses the HTML document to produce a simplified document tree structure. Second, the NLP pipeline takes the document tree and labels sentences with linguistic labels. Third, the labeled sentences are annotated by the annotators to produce a phrase graph containing all the annotations. Finally, the graph building stage deploys term normalization and purpose classification to transform the phrase graph into a PoliGraph.
  • Figure 5: The PoliGraph generated from 'Puzzle 100 Doors" app's privacy policy.
  • ...and 6 more figures