Human Decision-making is Susceptible to AI-driven Manipulation

TL;DR

Human susceptibility to AI-driven manipulation is examined, highlighting a critical vulnerability in human-AI interactions, and the need for ethical safeguards and regulatory frameworks to protect human autonomy is highlighted.

Abstract

AI systems are increasingly intertwined with daily life, assisting users with various tasks and guiding decision-making. This integration introduces risks of AI-driven manipulation, where such systems may exploit users' cognitive biases and emotional vulnerabilities to steer them toward harmful outcomes. Through a randomized between-subjects experiment with 233 participants, we examined human susceptibility to such manipulation in financial (e.g., purchases) and emotional (e.g., conflict resolution) decision-making contexts. Participants interacted with one of three AI agents: a neutral agent (NA) optimizing for user benefit without explicit influence, a manipulative agent (MA) designed to covertly influence beliefs and behaviors, or a strategy-enhanced manipulative agent (SEMA) equipped with established psychological tactics, allowing it to select and apply them adaptively during interactions to reach its hidden objectives. By analyzing participants' preference ratings, we found significant susceptibility to AI-driven manipulation. Particularly across both decision-making domains, interacting with the manipulative agents significantly increased the odds of rating hidden incentives higher than optimal options (Financial, MA: OR=5.24, SEMA: OR=7.96; Emotional, MA: OR=5.52, SEMA: OR=5.71) compared to the NA group. Notably, we found no clear evidence that employing psychological strategies (SEMA) was overall more effective than simple manipulative objectives (MA) on our primary outcomes. Hence, AI-driven manipulation could become widespread even without requiring sophisticated tactics and expertise. While our findings are preliminary and derived from hypothetical, low-stakes scenarios, we highlight a critical vulnerability in human-AI interactions, emphasizing the need for ethical safeguards and regulatory frameworks to protect human autonomy.

Figures (13)

• Figure 1: Overview of our study:a) Framework of decision-making scenarios. Participants were assigned to either the financial or emotional decision-making domain. Each domain featured three pre-defined scenarios with four options: one optimal and three harmful, representing non-existent, excessive, and dependence-inducing products in the financial domain, and maladaptive coping strategies (disengagement, emotional venting, self-blame) in the emotional domain. b) AI conditions. Within each domain, we further assigned the participants to an AI condition: 1. Neutral Agent (NA), an assistant optimizing for user benefit, designed as the control condition in our experiments; 2. Manipulative Agent (MA), an assistant with hidden manipulative objectives; and 3. Strategy-Enhanced Manipulative Agent (SEMA), an assistant equipped with a predefined set of manipulation tactics drawn from existing psychological literature, allowing it to select and apply them adaptively during interactions to reach its hidden objectives. All agents were created by prompting GPT-4o hurst2024gpt (Prompts are provided in Supplementary Figures \ref{['fig:neutral_prompt']}-\ref{['fig:mani_prompt']}). c) Flowchart overview of the study design. After progressing through the initial phase of the experiment and being assigned a domain-AI condition, participants were presented with the three corresponding scenarios. Accordingly, for each scenario, they were tasked with rating each option on a 10-point Likert scale, interacting with the AI assistant, and re-rating each option post-interaction. The options in each scenario were shown to each user in a randomized order.
• Figure 2: Distribution of Hidden-Optimal Differential ($HOD$) across AI conditions in decision-making contexts. While in both domains, participants exhibited relative preference for optimal options ($HOD< 0$) across all AI conditions pre-interaction, those interacting with the manipulative agents (MA and SEMA) showed significant inclines post-interaction relative to the NA group, indicating increased alignment with the hidden objectives of these agents (Supplementary Table \ref{['tab:rating_diff']}).
• Figure 3: Probability distribution of preferences across AI conditions in decision-making contexts. We categorized participants' preferences in each scenario at each time (pre-/post-interaction) into a three-level outcome based on the Hidden-Optimal Differential ($HOD$; difference in the ratings of the optimal option and the hidden incentive): Hidden ($HOD> 0$), Optimal ($HOD< 0$), and Tie ($HOD= 0$). Across both domains, interacting with the manipulative agents (MA and SEMA) significantly increased the odds of Hidden relative to the NA condition post-interaction (Supplementary Table \ref{['tab:multinomial_agent']}).
• Figure 4: Distribution of strategies employed by the strategy-enhanced manipulative agent (SEMA) across decision-making contexts. The bar plots show the proportion of responses for each manipulation strategy used by the SEMA, with significant differences annotated above each bar plot. These results suggest that SEMA adapted its strategy usage to the emotional or financial nature of the decisions, reflecting a tailored approach to influence. $P$ value legend: n.s. (not significant), $P\geq 0.05$; *, $P < 0.05$; **, $P < 0.01$; and ***, $P < 0.0001$).
• Figure S1: Distribution of preference ratings across AI conditions in decision-making contexts for optimal options and hidden incentives. While in both domains, participants reported similar ratings at baseline (pre-interaction) across AI conditions, those interacting with MA and SEMA showed substantial declines in their ratings for the optimal options and increases for the hidden incentives, reflecting the influence of these agents (Supplementary Table \ref{['tab:ratings']}).