Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PRISMe: A Novel LLM-Powered Tool for Interactive Privacy Policy Assessment

Vincent Freiberger, Arthur Fleig, Erik Buchmann

TL;DR

PRISMe tackles the problem of unreadable privacy policies by introducing an LLM-driven privacy policy assessment tool that combines a dynamic, context-aware evaluation dashboard with a chat interface. The Chrome extension scrapes plain-text policies, analyzes them with GPT-4o to yield per-criterion ratings, and presents results via an overview, interactive dashboard, and conversation, enabling customized explanations. In a mixed-methods study with 22 participants, PRISMe improved understanding and privacy awareness while highlighting issues in consistency, trust calibration, and potential LLM inaccuracies, informing design implications for future policy-analysis tools. Overall, PRISMe demonstrates a promising direction for user-centered, AI-assisted privacy policy comprehension and offers actionable guidance to enhance accessibility, comparability, and reliability of policy assessments.

Abstract

Protecting online privacy requires users to engage with and comprehend website privacy policies, but many policies are difficult and tedious to read. We present PRISMe (Privacy Risk Information Scanner for Me), a novel Large Language Model (LLM)-driven privacy policy assessment tool, which helps users to understand the essence of a lengthy, complex privacy policy while browsing. The tool, a browser extension, integrates a dashboard and an LLM chat. One major contribution is the first rigorous evaluation of such a tool. In a mixed-methods user study (N=22), we evaluate PRISMe's efficiency, usability, understandability of the provided information, and impacts on awareness. While our tool improves privacy awareness by providing a comprehensible quick overview and a quality chat for in-depth discussion, users note issues with consistency and building trust in the tool. From our insights, we derive important design implications to guide future policy analysis tools.

PRISMe: A Novel LLM-Powered Tool for Interactive Privacy Policy Assessment

TL;DR

PRISMe tackles the problem of unreadable privacy policies by introducing an LLM-driven privacy policy assessment tool that combines a dynamic, context-aware evaluation dashboard with a chat interface. The Chrome extension scrapes plain-text policies, analyzes them with GPT-4o to yield per-criterion ratings, and presents results via an overview, interactive dashboard, and conversation, enabling customized explanations. In a mixed-methods study with 22 participants, PRISMe improved understanding and privacy awareness while highlighting issues in consistency, trust calibration, and potential LLM inaccuracies, informing design implications for future policy-analysis tools. Overall, PRISMe demonstrates a promising direction for user-centered, AI-assisted privacy policy comprehension and offers actionable guidance to enhance accessibility, comparability, and reliability of policy assessments.

Abstract

Protecting online privacy requires users to engage with and comprehend website privacy policies, but many policies are difficult and tedious to read. We present PRISMe (Privacy Risk Information Scanner for Me), a novel Large Language Model (LLM)-driven privacy policy assessment tool, which helps users to understand the essence of a lengthy, complex privacy policy while browsing. The tool, a browser extension, integrates a dashboard and an LLM chat. One major contribution is the first rigorous evaluation of such a tool. In a mixed-methods user study (N=22), we evaluate PRISMe's efficiency, usability, understandability of the provided information, and impacts on awareness. While our tool improves privacy awareness by providing a comprehensible quick overview and a quality chat for in-depth discussion, users note issues with consistency and building trust in the tool. From our insights, we derive important design implications to guide future policy analysis tools.

Paper Structure

This paper contains 64 sections, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Challenges with Privacy Policies
  4. The Landscape of Transparency-Enhancing Technologies
  5. Approaches reliant on a privacy language representation of the policy
  6. Reliance on processed data
  7. Automated assessment with Natural Language Processing (NLP)
  8. LLM-based privacy policy assessment
  9. ML classifiers
  10. Shorter and tailored privacy policies
  11. TETs beyond privacy policy analysis and tools for service providers
  12. Identified Gaps and PRISMe's Position in the TET Landscape
  13. Design Process -- Our Road to PRISMe
  14. Facet 1: Scrollbar Feedback and Policy Visibility
  15. Facet 2: Chat-Based Interactive Exploration
  16. ...and 49 more sections

Figures (8)

  • Figure 1: Possible usage flow of PRISMe: The user first visits a website. Our tool features privacy alerts via colored scrollbars and a point-of-entry smiley icon (top left). Clicking the smiley opens an Overview Panel (top right) summarizing key privacy issues, with navigation to a Dynamic Dashboard and chat interface. The dashboard (bottom right) provides detailed policy evaluation criteria, which allows to go into a chat interface (bottom left) by clicking more below a criterion.
  • Figure 2: Participant's age distribution, education level, and self-assessed confidence in understanding privacy policies
  • Figure 3: Overview of the procedure of the study for each participant
  • Figure 4: Time spent by participants on the different screens of the tool
  • Figure 5: Usage frequency of suggestions and questions that participants came up with on their own labeled by participant
  • ...and 3 more figures