No More Trade-Offs. GPT and Fully Informative Privacy Policies

TL;DR

Problem: privacy policies are either comprehensive but hard to understand or concise but incomplete; the paper proposes fully comprehensive privacy policies and argues for legal adoption. Approach: introduce a mock Orderoo Inc privacy policy in a fully comprehensive format and evaluate ChatGPT-3.5 and -4 on six questions with short and long prompts. Findings: GPT-4 can answer most questions correctly from a fully comprehensive policy, while GPT-3.5 struggles, with few false positives and some false negatives; longer prompts may reduce accuracy for GPT-4 and have little effect on GPT-3.5. Implications: automated QA over comprehensive policies could enable meaningful user insight, supporting regulatory reform toward comprehensive disclosures, albeit with need for post processing and longer policies. Future work: expand documents, test additional LLMs, develop automated controls, and integrate text linked QA.

Abstract

The paper reports the results of an experiment aimed at testing to what extent ChatGPT 3.5 and 4 is able to answer questions regarding privacy policies designed in the new format that we propose. In a world of human-only interpreters, there was a trade-off between comprehensiveness and comprehensibility of privacy policies, leading to the actual policies not containing enough information for users to learn anything meaningful. Having shown that GPT performs relatively well with the new format, we provide experimental evidence supporting our policy suggestion, namely that the law should require fully comprehensive privacy policies, even if this means they become less concise.

Figures (1)

• Figure 1: Mock privacy policy