Empowering Users in Digital Privacy Management through Interactive LLM-Based Agents

TL;DR

It is demonstrated that LLMs significantly outperform traditional models in tasks like Data Practice Identification, Choice Identification, Policy Summarization, and Privacy Question Answering, setting new benchmarks in privacy policy analysis.

Abstract

This paper presents a novel application of large language models (LLMs) to enhance user comprehension of privacy policies through an interactive dialogue agent. We demonstrate that LLMs significantly outperform traditional models in tasks like Data Practice Identification, Choice Identification, Policy Summarization, and Privacy Question Answering, setting new benchmarks in privacy policy analysis. Building on these findings, we introduce an innovative LLM-based agent that functions as an expert system for processing website privacy policies, guiding users through complex legal language without requiring them to pose specific questions. A user study with 100 participants showed that users assisted by the agent had higher comprehension levels (mean score of 2.6 out of 3 vs. 1.8 in the control group), reduced cognitive load (task difficulty ratings of 3.2 out of 10 vs. 7.8), increased confidence in managing privacy, and completed tasks in less time (5.5 minutes vs. 15.8 minutes). This work highlights the potential of LLM-based agents to transform user interaction with privacy policies, leading to more informed consent and empowering users in the digital services landscape.

Figures (7)

• Figure 1: Overview of the workflow for benchmarking large language models on usable privacy policy tasks and designing a guided agent to assist consumers.
• Figure 2: The figure illustrates the development workflow of this agent. The system integrates multiple tools to automate the extraction and interpretation of privacy practices, providing users with summarized insights and question-answering capabilities.
• Figure 3: The initial screen of the Privacy Policy Analysis Agent, prompting users to enter a URL for detailed policy analysis.
• Figure 4: Comprehensive breakdown of the IMDb Privacy Notice, showing various sections and the extent of information provided.
• Figure 5: Detailed explanation of the third-party sharing and collection practices outlined in the IMDb Privacy Notice.