Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adversarial Negotiation Dynamics in Generative Language Models

Arinbjörn Kolbeinsson, Benedikt Kolbeinsson

TL;DR

The paper investigates how different language agents perform in adversarial contract negotiations, addressing robustness and safety when opponents are unknown. It uses head-to-head competitions among eight models, with contracts generated and amended under constrained prompts and judged by a panel of six peers. Key findings show opponent-dependent performance: general-purpose models can outperform specialized ones in some roles while being vulnerable to others, and role biases emerge (e.g., Phi-3 seller-skew). The work highlights safety and fairness considerations in high-stakes legal contexts and provides practical guidance on model selection and red-teaming strategies to improve reliability and risk mitigation.

Abstract

Generative language models are increasingly used for contract drafting and enhancement, creating a scenario where competing parties deploy different language models against each other. This introduces not only a game-theory challenge but also significant concerns related to AI safety and security, as the language model employed by the opposing party can be unknown. These competitive interactions can be seen as adversarial testing grounds, where models are effectively red-teamed to expose vulnerabilities such as generating biased, harmful or legally problematic text. Despite the importance of these challenges, the competitive robustness and safety of these models in adversarial settings remain poorly understood. In this small study, we approach this problem by evaluating the performance and vulnerabilities of major open-source language models in head-to-head competitions, simulating real-world contract negotiations. We further explore how these adversarial interactions can reveal potential risks, informing the development of more secure and reliable models. Our findings contribute to the growing body of research on AI safety, offering insights into model selection and optimisation in competitive legal contexts and providing actionable strategies for mitigating risks.

Adversarial Negotiation Dynamics in Generative Language Models

TL;DR

The paper investigates how different language agents perform in adversarial contract negotiations, addressing robustness and safety when opponents are unknown. It uses head-to-head competitions among eight models, with contracts generated and amended under constrained prompts and judged by a panel of six peers. Key findings show opponent-dependent performance: general-purpose models can outperform specialized ones in some roles while being vulnerable to others, and role biases emerge (e.g., Phi-3 seller-skew). The work highlights safety and fairness considerations in high-stakes legal contexts and provides practical guidance on model selection and red-teaming strategies to improve reliability and risk mitigation.

Abstract

Generative language models are increasingly used for contract drafting and enhancement, creating a scenario where competing parties deploy different language models against each other. This introduces not only a game-theory challenge but also significant concerns related to AI safety and security, as the language model employed by the opposing party can be unknown. These competitive interactions can be seen as adversarial testing grounds, where models are effectively red-teamed to expose vulnerabilities such as generating biased, harmful or legally problematic text. Despite the importance of these challenges, the competitive robustness and safety of these models in adversarial settings remain poorly understood. In this small study, we approach this problem by evaluating the performance and vulnerabilities of major open-source language models in head-to-head competitions, simulating real-world contract negotiations. We further explore how these adversarial interactions can reveal potential risks, informing the development of more secure and reliable models. Our findings contribute to the growing body of research on AI safety, offering insights into model selection and optimisation in competitive legal contexts and providing actionable strategies for mitigating risks.
Paper Structure (11 sections, 2 figures)

This paper contains 11 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Experimental Setup
  3. Contract Generation and Amendments
  4. Judgement Process
  5. Model Pairings and Matchups
  6. Models Used
  7. Fairness, Safety and Red Teaming Considerations
  8. Results and Discussion
  9. Game-Theoretic Dynamics and Strategic Model Selection
  10. Limitations and Future Work
  11. Conclusion

Figures (2)

  • Figure 1: Scatter plot depicting the head-to-head competition outcomes between different language models acting as either sellers or buyers in contract negotiations. The x-axis represents the number of seller wins, and the y-axis represents the number of buyer wins. Each point corresponds to a specific language model. The Llama models, Llama-3 and Llama-2, show excellent buyer and seller performance, respectively, while models like Gemma and Mistral exhibit a more balanced performance between roles.
  • Figure 2: Normalised win differences between pairs of language models acting as buyer and seller agents in simulated contract negotiations. Each cell value is the normalised difference between the number of judges favouring the seller agent (positive, blue) and those favouring the buyer agent (negative, orange), centred at zero (equal wins).