Do LLMs trust AI regulation? Emerging behaviour of game-theoretic LLM agents

TL;DR

The paper investigates how trust and regulation emerge in AI governance by embedding LLM agents (GPT-4o and Mistral Large) into a three-player evolutionary game with Users, Developers, and Regulators under conditional and unconditional trust. Using FAIRGAME to instantiate one-shot and repeated interactions, it reveals that LLMs produce nuanced, model-dependent behaviours that often deviate from pure game-theoretic predictions, with conditional trust sometimes undermining cooperation yet full user trust boosting virtuous regulation dynamics. Key contributions include evidence that reputation incentives and trust dynamics interact with model biases to shape safety-oriented outcomes, and a demonstration of how LLMs can be used to explore governance scenarios and predict potential regulatory outcomes. The work highlights the practical significance of careful LLM selection, transparency, and the integration of game-theoretic insights to inform AI regulation design and anticipate strategic behaviours of AI agents in governance tasks.

Abstract

There is general agreement that fostering trust and cooperation within the AI development ecosystem is essential to promote the adoption of trustworthy AI systems. By embedding Large Language Model (LLM) agents within an evolutionary game-theoretic framework, this paper investigates the complex interplay between AI developers, regulators and users, modelling their strategic choices under different regulatory scenarios. Evolutionary game theory (EGT) is used to quantitatively model the dilemmas faced by each actor, and LLMs provide additional degrees of complexity and nuances and enable repeated games and incorporation of personality traits. Our research identifies emerging behaviours of strategic AI agents, which tend to adopt more "pessimistic" (not trusting and defective) stances than pure game-theoretic agents. We observe that, in case of full trust by users, incentives are effective to promote effective regulation; however, conditional trust may deteriorate the "social pact". Establishing a virtuous feedback between users' trust and regulators' reputation thus appears to be key to nudge developers towards creating safe AI. However, the level at which this trust emerges may depend on the specific LLM used for testing. Our results thus provide guidance for AI regulation systems, and help predict the outcome of strategic LLM agents, should they be used to aid regulation itself.

Figures (7)

• Figure 1: Scheme of the core features for the three-player interaction model. Users may trust or not; if they do ($T$), other agents get benefits (blue lines). If they do not ($N$), no adoption is enacted and other agents get no benefits. Developers may comply ($C$) with regulations and develop safe AI which, if adopted, yields benefits for users (green line); however, it may be costly. Instead, developers may create unsafe AI ($D$) which, if adopted, may yield partial or negative payoff to users (red line). Regulators may be strict ($C$), using resources but gaining benefits if they catch defective developers (a), or lenient ($D$). If users have access to regulators' reputation (b), they can decide whether to trust conditionally.
• Figure 2: Results for the one-shot game, using GPT-4o. Left box: low regulation cost ($c_R$ = 0.5). Right box: high regulation cost ($c_R = 5$). Each panel corresponds to a different value for $\epsilon$, i.e., the risk for users to adopt unsafe AI ($\epsilon<0$ has higher risk). Conditional trust promotes full trust, cooperative regulation and safe development. Parameters set to: $b_U$ = $b_R$ = $b_P$ = 4, $u$ = 1.5, $v$ = 0.5, $c_P$ = 0.5.
• Figure 3: Results for the one-shot game, using Mistral Large. Left box: low regulation cost ($c_R$ = 0.5). Right box: high regulation cost ($c_R = 5$). Each panel corresponds to a different value for $\epsilon$, i.e., the risk for users to adopt unsafe AI ($\epsilon<0$ has higher risk). Conditional trust yields low trust. Parameters set to: $b_U$ = $b_R$ = $b_P$ = 4, $u$ = 1.5, $v$ = 0.5, $c_P$ = 0.5.
• Figure 4: Results for the repeated games over 10 rounds (average over rounds), using GPT-4o and Mistral Large. Left box: low regulation cost ($c_R$ = 0.5). Right box: high regulation cost ($c_R = 5$). Each panel corresponds to a different value for $\epsilon$, i.e., the risk for users to adopt unsafe AI ($\epsilon<0$ has higher risk).
• Figure 5: Results for the repeated games over each of the 10 rounds, for different values of $b_{fo}$, using GPT-4o. Left box: low regulation cost ($c_R$ = 0.5). Right box: high regulation cost ($c_R = 5$). Each panel corresponds to a different value for $\epsilon$, i.e., the risk for users to adopt unsafe AI ($\epsilon<0$ has higher risk).