Autonomy Matters: A Study on Personalization-Privacy Dilemma in LLM Agents

TL;DR

This study investigates the personalization-privacy dilemma in LLM agents by manipulating three personalization types (Basic, Privacy-Aware, No Personalization) and three autonomy levels (No, Intermediate, Full) in a $3\times3$ between-subjects design with $N=450$. It finds that personalization aligned with privacy preferences reduces privacy concerns and enhances trust and willingness to use, while Basic personalization increases privacy concerns and lowers trust. Crucially, Intermediate Autonomy attenuates these effects, likely by increasing perceived control, and mediates the relationship between personalization and outcomes through perceived sensitivity, control, and usefulness. The results imply that achieving perfect alignment is challenging, but designing agent autonomy to preserve meaningful human oversight offers a practical path to mitigate privacy risks and improve user acceptance and trust in agentic AI systems. These insights inform design guidelines for deploying LLM agents in daily tasks while safeguarding user privacy and autonomy.

Abstract

Large Language Model (LLM) agents require personal information for personalization in order to better act on users' behalf in daily tasks, but this raises privacy concerns and a personalization-privacy dilemma. Agent's autonomy introduces both risks and opportunities, yet its effects remain unclear. To better understand this, we conducted a 3$\times$3 between-subjects experiment ($N=450$) to study how agent's autonomy level and personalization influence users' privacy concerns, trust and willingness to use, as well as the underlying psychological processes. We find that personalization without considering users' privacy preferences increases privacy concerns and decreases trust and willingness to use. Autonomy moderates these effects: Intermediate autonomy flattens the impact of personalization compared to No- and Full autonomy conditions. Our results suggest that rather than aiming for perfect model alignment in output generation, balancing autonomy of agent's action and user control offers a promising path to mitigate the personalization-privacy dilemma.

Figures (5)

• Figure 1: Overview of the study procedure with four steps: (1) Participants provided both non-sensitive and sensitive information for personalizing the LLM agent. (2) Participants were introduced to one of the nine LLM agent conditions. (3) Participants used the assigned LLM agent in the communication task. A sensitivity detection module monitored the agent’s generated messages and provided reminders based on autonomy levels (the Intermediate Autonomy condition is shown as an example). (4) Participants reported their perceptions of the LLM agent.
• Figure 2: Estimates of people's (a) privacy concern, (b) trust, and (c) willingness to use across nine experimental conditions (3 personalization types $\times$ 3 autonomy levels).
• Figure 3: Significant interaction effects of personalization type and agent autonomy level on (a) privacy concern, (b) trust, and (c) willingness to use. Points represent estimated marginal means from the mixed-effects model, and vertical bars indicate 95% confidence intervals of the estimated ratings.
• Figure 4: Estimates of the percentage of people who (a) perceived sensitivity (answered "Yes" in the question about whether they thought LLM agent’s responses contained any sensitive information), (b) perceived control (mean control rating > 3), and (c) perceived usefulness (mean usefulness rating > 3) across nine experimental conditions (3 personalization types $\times$ 3 autonomy levels).
• Figure 5: Moderated mediation model tested for H5 and H6. Personalization (No personalization, Privacy-aware personalization; Basic personalization as reference) is modeled as $X$, Autonomy (Intermediate autonomy and Full autonomy; No autonomy as reference) as $W$, and their interaction ($X\times W$). Perceived Sensitivity (M1), Perceived Control (M2), and Perceived Usefulness (M3) serve as mediators, with Privacy Concern, Trust, and Willingness to Use as outcomes ($Y$). The model tests both mediation pathways ($a \times b$) and the moderation of personalization effects by autonomy.