Adversarial Testing in LLMs: Insights into Decision-Making Vulnerabilities

TL;DR

The paper proposes an adversarial evaluation framework to diagnose LLM decision-making vulnerabilities by modeling LLMs as learners manipulated by a reinforcement-learning adversary. It implements a learner model as an RNN that predicts next actions from past actions, rewards, and observations, and trains an adversary via Deep Q-learning to perturb the environment. The framework is applied to two canonical tasks—the two-armed bandit and the Multi-Round Trust Task—across GPT-3.5, GPT-4, Gemini-1.5, and DeepSeek-V3, revealing model-specific patterns of exploitation, rigidity, and adaptability. The findings illustrate how adversarial strategies can steer decisions, underscore the importance of fairness recognition and flexible strategy use, and position the framework as a diagnostic tool for AI safety and alignment research.

Abstract

As Large Language Models (LLMs) become increasingly integrated into real-world decision-making systems, understanding their behavioural vulnerabilities remains a critical challenge for AI safety and alignment. While existing evaluation metrics focus primarily on reasoning accuracy or factual correctness, they often overlook whether LLMs are robust to adversarial manipulation or capable of using adaptive strategy in dynamic environments. This paper introduces an adversarial evaluation framework designed to systematically stress-test the decision-making processes of LLMs under interactive and adversarial conditions. Drawing on methodologies from cognitive psychology and game theory, our framework probes how models respond in two canonical tasks: the two-armed bandit task and the Multi-Round Trust Task. These tasks capture key aspects of exploration-exploitation trade-offs, social cooperation, and strategic flexibility. We apply this framework to several state-of-the-art LLMs, including GPT-3.5, GPT-4, Gemini-1.5, and DeepSeek-V3, revealing model-specific susceptibilities to manipulation and rigidity in strategy adaptation. Our findings highlight distinct behavioral patterns across models and emphasize the importance of adaptability and fairness recognition for trustworthy AI deployment. Rather than offering a performance benchmark, this work proposes a methodology for diagnosing decision-making weaknesses in LLM-based agents, providing actionable insights for alignment and safety research.

Figures (6)

• Figure 1: The adversarial framework (adapted from dezfouli2020adversarial). (A) The interaction of the LLM with the task. Each simulation cycle begins with the LLM receiving a learner reward ($r_{t-1}^n$) for the prior action along with a new observation ($o_t^n$) from the environment. Based on this, the GPT executes an action ($a_t^n$), and the cycle repeats with the environment providing updated rewards and observations. (B) The LLM's actions are modelled by a RNN with parameters $\Theta$. Inputs to the RNN include the previous action ($a_{t-1}^n$), the most recent learner rewards ($r_{t-1}^n$), and the current observations ($o_t^n$), along with the RNN's last internal state ($x_{t-1}^n$). After receiving the inputs, the RNN updates its internal state and predicts the next action using a softmax layer ($\pi_t^n$). These predictions are then compared with the actual actions taken by the LLM and evaluated with a loss function ($\mathscr{L}(\Theta)$) in order to train the model. The trained model is called the learner model. (C) The adversary is an RL agent, which is trained to manipulate the decision-making environment of the learner model. Utilizing the latest internal state ($x_t^n$) of the learner model, which encapsulates its cumulative learning experiences, the adversary determines the learner reward ($r_t^n$) and the next observation ($o_{t+1}^n$) to be delivered to the learner model. This strategic input is designed to steer the learner model's subsequent actions ($a_t^n$) toward achieving the adversary’s predefined objectives. The adversarial reward (Reward), which is used to train the adversary, depends on the alignment between the action taken by the learner model ($a_t^n$) and the adversary’s objectives. (D) Using the trained adversary and the learner model for generating adversarial interactions with the LLM. In each simulation $n$, the LLM processes the rewards ($r_{t-1}^n$) and observations ($o_t^n$) from the adversary, responding with actions ($a_t^n$) that update the learner model’s internal state ($x_t^n$). This state is then sent to the adversary to determine the learner's reward for the action ($a_t^n$) and the next observation ($o_{t+1}^n$). This cycle continues till the end of the task.
• Figure 2: A: Example prompt for one trial in the bandit task for the LLMs. B: Behavioural pattern in trials of three random human participants and three sample simulations for each of the LLMs.
• Figure 3: A: LLMs' behaviour compared to human behaviour on average of each simulation (or individual), measured by reward rate, percentage choosing the target option, no-reward-switch rate, and reward-switch rate. B: The performance of human and the LLMs, which is measured by the percentage of the target action selection before and after adversarial influence.
• Figure 4: Four sample simulations of the trained adversaries against four LLMs. The plot presents the strategies used by the adversaries and the responses of the LLMs. (A) Adversary versus GPT-3.5. (B) Adversary versus GPT-4. (C) Adversary versus Gemini-1.5 (D) Adversary versus DeepSeek-V3
• Figure 5: A: Example prompt for one round in the MRTT as presented to the LLMs. The model must decide how much to invest given previous outcomes. B: Average investment behaviour of humans and LLMs when playing against a random trustee in the MRTT. Investment amounts are plotted as a function of the repayment proportion in the previous round. C: Total earnings of the trustee (the adversary) and the investor (human or LLM), along with the absolute earning gap across different adversarial conditions: Random (RND), Fair (FAIR), and Maximizing (MAX). Results are shown for humans and each LLM.