Position: Contextual Integrity is Inadequately Applied to Language Models
Yan Shvartzshnaider, Vasisht Duddu
TL;DR
This paper argues that Contextual Integrity (CI) offers a robust privacy framework for evaluating LLMs, but current literature fails to faithfully implement CI’s core tenets. It clarifies the four tenets, systematizes nine prior CI-based studies into alignment or deviation, and highlights experimental hygiene issues such as prompt sensitivity and position bias that threaten non-adversarial rigor. The authors demonstrate that many works conflate CI with legal statutes, data minimization, or crowd preferences, leading to potentially flawed privacy-preserving designs. They advocate a multidisciplinary, governance-informed approach to CI in LLMs and provide concrete recommendations to implement CI tenets and enhance the reliability of normative evaluations.
Abstract
Machine learning community is discovering Contextual Integrity (CI) as a useful framework to assess the privacy implications of large language models (LLMs). This is an encouraging development. The CI theory emphasizes sharing information in accordance with privacy norms and can bridge the social, legal, political, and technical aspects essential for evaluating privacy in LLMs. However, this is also a good point to reflect on use of CI for LLMs. This position paper argues that existing literature inadequately applies CI for LLMs without embracing the theory's fundamental tenets. Inadequate applications of CI could lead to incorrect conclusions and flawed privacy-preserving designs. We clarify the four fundamental tenets of CI theory, systematize prior work on whether they deviate from these tenets, and highlight overlooked issues in experimental hygiene for LLMs (e.g., prompt sensitivity, positional bias).