Better Zero-Shot Reasoning with Role-Play Prompting

TL;DR

The paper investigates whether role-playing can boost zero-shot reasoning in large language models. It introduces a two-stage Role-Play Prompting framework that first constructs task-specific role-immersion prompts and then elicits answers under that role in a focused two-round process. Evaluations across 12 reasoning benchmarks show broad improvements over standard zero-shot and often over Zero-Shot-CoT, indicating that role-playing acts as an implicit CoT trigger. The findings emphasize careful prompt design, including immersion and role advantages, and demonstrate generalization across models and scales, with public code released for replication.

Abstract

Modern large language models (LLMs) exhibit a remarkable capacity for role-playing, enabling them to embody not only human characters but also non-human entities. This versatility allows them to simulate complex human-like interactions and behaviors within various contexts, as well as to emulate specific objects or systems. While these capabilities have enhanced user engagement and introduced novel modes of interaction, the influence of role-playing on LLMs' reasoning abilities remains underexplored. In this study, we introduce a strategically designed role-play prompting methodology and assess its performance under the zero-shot setting across twelve diverse reasoning benchmarks. Our empirical results illustrate that role-play prompting consistently surpasses the standard zero-shot approach across most datasets. Notably, in experiments conducted using ChatGPT, accuracy on AQuA rises from 53.5% to 63.8%, and on Last Letter from 23.8% to 84.2%.Upon further comparison with the Zero-Shot-CoT technique, which prompts the model to "think step by step", our study demonstrates that role-play prompting acts as a more effective trigger for the CoT process. This highlights its potential to augment the reasoning capabilities of LLMs. We release our code at https://github.com/NKU-HLT/Role-Play-Prompting.

Figures (5)

• Figure 1: Examples of ChatGPT with (a) zero-shot and (b) role-play prompting. The role-play prompts are highlighted.
• Figure 2: The two-stage framework of our proposed role-play prompting. The role-play prompts are highlighted.
• Figure 3: An illustration of the two-stage role-play prompting procedure, exemplified with the commonsense reasoning task. In stage 1, multiple role-feedback prompts are sampled. In stage 2, the optimal role-feedback prompt (underlined in blue) is selected for answer generation.
• Figure 4: Accuracy comparison of Role-Play Prompting across different sizes of Llama 2-Chat models. See more details in Appendix \ref{['app: model_scale']}.
• Figure 5: A sketch map of answer extraction for role-play prompting.