Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Role-Play Paradox in Large Language Models: Reasoning Performance Gains and Ethical Dilemmas

Jinman Zhao, Zifan Qian, Linbo Cao, Yining Wang, Yitian Ding, Yulan Hu, Zeyu Zhang, Zeyong Jin

TL;DR

This paper investigates the dual nature of role-play in large language models (LLMs): it can enhance reasoning and contextually relevant outputs but also risks producing harmful or biased content. The authors deploy stereotype/toxicity benchmarks CrowS-Pairs, StereoSet, and HarmfulQ, and test direct role-play prompts as well as an autonomous role-autotuning framework across multiple LLMs (e.g., GPT-3.5, GPT-4o, Mixtral-8x7B). They find that role-play amplifies biases and toxic outputs across most roles, including neutral ones, and that automated role selection can significantly degrade accuracy while increasing unsafe responses. The study highlights the need for robust mitigation strategies, such as bias detection, adversarial training, and adaptive role frameworks, and suggests expanding evaluations to multilingual and cross-cultural contexts to ensure safe, trustworthy deployment.

Abstract

Role-play in large language models (LLMs) enhances their ability to generate contextually relevant and high-quality responses by simulating diverse cognitive perspectives. However, our study identifies significant risks associated with this technique. First, we demonstrate that autotuning, a method used to auto-select models' roles based on the question, can lead to the generation of harmful outputs, even when the model is tasked with adopting neutral roles. Second, we investigate how different roles affect the likelihood of generating biased or harmful content. Through testing on benchmarks containing stereotypical and harmful questions, we find that role-play consistently amplifies the risk of biased outputs. Our results underscore the need for careful consideration of both role simulation and tuning processes when deploying LLMs in sensitive or high-stakes contexts.

Role-Play Paradox in Large Language Models: Reasoning Performance Gains and Ethical Dilemmas

TL;DR

This paper investigates the dual nature of role-play in large language models (LLMs): it can enhance reasoning and contextually relevant outputs but also risks producing harmful or biased content. The authors deploy stereotype/toxicity benchmarks CrowS-Pairs, StereoSet, and HarmfulQ, and test direct role-play prompts as well as an autonomous role-autotuning framework across multiple LLMs (e.g., GPT-3.5, GPT-4o, Mixtral-8x7B). They find that role-play amplifies biases and toxic outputs across most roles, including neutral ones, and that automated role selection can significantly degrade accuracy while increasing unsafe responses. The study highlights the need for robust mitigation strategies, such as bias detection, adversarial training, and adaptive role frameworks, and suggests expanding evaluations to multilingual and cross-cultural contexts to ensure safe, trustworthy deployment.

Abstract

Role-play in large language models (LLMs) enhances their ability to generate contextually relevant and high-quality responses by simulating diverse cognitive perspectives. However, our study identifies significant risks associated with this technique. First, we demonstrate that autotuning, a method used to auto-select models' roles based on the question, can lead to the generation of harmful outputs, even when the model is tasked with adopting neutral roles. Second, we investigate how different roles affect the likelihood of generating biased or harmful content. Through testing on benchmarks containing stereotypical and harmful questions, we find that role-play consistently amplifies the risk of biased outputs. Our results underscore the need for careful consideration of both role simulation and tuning processes when deploying LLMs in sensitive or high-stakes contexts.
Paper Structure (17 sections, 7 figures, 3 tables)

This paper contains 17 sections, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Role-Play in LLM
  4. Bias, Stereotypes, and Toxicity in AI
  5. Method
  6. Stereotype and Toxicity Evaluation
  7. Direct Role Play
  8. Role Autotune
  9. Experimental Setup
  10. Model Selection
  11. Hyperparameter Setting
  12. Main Results
  13. Direct Role Play
  14. Role Autotune
  15. Other Study
  16. ...and 2 more sections

Figures (7)

  • Figure 1: Example of ChatGPT with a question that is harmful.
  • Figure 2: Role auto tune example.
  • Figure 3: Bias accuracy of different roles among CrowS-Pairs, using GPT3.5 and Mixtral-8x7B. The baseline represents no role assigned and use the basic prompt.
  • Figure 4: Bias accuracy of different roles among Stereoset, using GPT3.5 and Mixtral-8x7B. The baseline represents no role assigned and use the basic prompt.
  • Figure 5: Result of Auto-tune Role on GPT-3.5 across two datasets via zero shot prompt.
  • ...and 2 more figures