Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

"You don't need a university degree to comprehend data protection this way": LLM-Powered Interactive Privacy Policy Assessment

Vincent Freiberger, Arthur Fleig, Erik Buchmann

TL;DR

Privacy policies are often dense and poorly understood by users, limiting awareness of data practices. The authors present PRISMe, an LLM-powered browser extension with an interactive dashboard and chat that assesses policies and communicates findings in plain language, evaluated through a qualitative lab study with 22 participants. Findings show improved understanding and increased privacy awareness, but also highlight trust issues and occasional LLM limitations, including hallucinations and incomplete policy scraping. The work provides design implications for interactive, evidence-backed privacy policy tools and suggests pathways for more reliable, user-centric AI-assisted policy analysis in real-world use.

Abstract

Protecting online privacy requires users to engage with and comprehend website privacy policies, but many policies are difficult and tedious to read. We present the first qualitative user study on Large Language Model (LLM)-driven privacy policy assessment. To this end, we build and evaluate an LLM-based privacy policy assessment browser extension, which helps users understand the essence of a lengthy, complex privacy policy while browsing. The tool integrates a dashboard and an LLM chat. In our qualitative user study (N=22), we evaluate usability, understandability of the information our tool provides, and its impacts on awareness. While providing a comprehensible quick overview and a chat for in-depth discussion improves privacy awareness, users note issues with building trust in the tool. From our insights, we derive important design implications to guide future policy analysis tools.

"You don't need a university degree to comprehend data protection this way": LLM-Powered Interactive Privacy Policy Assessment

TL;DR

Privacy policies are often dense and poorly understood by users, limiting awareness of data practices. The authors present PRISMe, an LLM-powered browser extension with an interactive dashboard and chat that assesses policies and communicates findings in plain language, evaluated through a qualitative lab study with 22 participants. Findings show improved understanding and increased privacy awareness, but also highlight trust issues and occasional LLM limitations, including hallucinations and incomplete policy scraping. The work provides design implications for interactive, evidence-backed privacy policy tools and suggests pathways for more reliable, user-centric AI-assisted policy analysis in real-world use.

Abstract

Protecting online privacy requires users to engage with and comprehend website privacy policies, but many policies are difficult and tedious to read. We present the first qualitative user study on Large Language Model (LLM)-driven privacy policy assessment. To this end, we build and evaluate an LLM-based privacy policy assessment browser extension, which helps users understand the essence of a lengthy, complex privacy policy while browsing. The tool integrates a dashboard and an LLM chat. In our qualitative user study (N=22), we evaluate usability, understandability of the information our tool provides, and its impacts on awareness. While providing a comprehensible quick overview and a chat for in-depth discussion improves privacy awareness, users note issues with building trust in the tool. From our insights, we derive important design implications to guide future policy analysis tools.

Paper Structure

This paper contains 29 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Challenges with Privacy Policies
  4. The Landscape of Transparency-Enhancing Technologies
  5. Identified gaps for AI tools assessing privacy policies
  6. Study Methodology
  7. Experimental Design and Procedure
  8. Participants
  9. Our PRISMe Prototype
  10. Results
  11. Interview & Comments Data
  12. Chat Responses
  13. Discussion
  14. Behavioral patterns
  15. \ref{['item:rq-understandable']}: How do users with varying privacy knowledge interpret PRISMe’s privacy policy explanations?
  16. ...and 14 more sections

Figures (3)

  • Figure 1: When the user visits a website, our prototype evaluates the privacy policy in the background and displays privacy alerts via colored scrollbars and a point-of-entry smiley icon (top middle). Clicking the smiley opens an Overview Panel (left) summarizing key privacy issues, with navigation to a Dynamic Dashboard and chat interface. The dashboard (bottom middle) provides detailed policy evaluation criteria, which users can chat about (right) by clicking the respective "More" button.
  • Figure 2: System Usability Scale results (questions with * have inverted scores; higher values are always better)
  • Figure 3: Results of our questions on a 5-point Likert scale (1: strongly disagree; 5: strongly agree).