When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models

TL;DR

This work systematically evaluates 162 personas across 4 open-source LLM families on 2,410 MMLU questions to determine whether social roles in system prompts enhance objective task performance. Using mixed-effects regression and a suite of persona attributes, it finds that personas rarely improve accuracy and can even hurt performance, with only modest benefits arising from domain-aligned or in-domain roles. It further investigates mechanisms via word frequency, prompt-question similarity, and perplexity, revealing only weak, model-dependent relationships, and shows automatic persona search yields limited gains. The study provides a practical pipeline and release of data and code to guide future system-prompt design and role-playing strategies, while highlighting the unpredictability of persona effects.

Abstract

Prompting serves as the major way humans interact with Large Language Models (LLM). Commercial AI systems commonly define the role of the LLM in system prompts. For example, ChatGPT uses ``You are a helpful assistant'' as part of its default system prompt. Despite current practices of adding personas to system prompts, it remains unclear how different personas affect a model's performance on objective tasks. In this study, we present a systematic evaluation of personas in system prompts. We curate a list of 162 roles covering 6 types of interpersonal relationships and 8 domains of expertise. Through extensive analysis of 4 popular families of LLMs and 2,410 factual questions, we demonstrate that adding personas in system prompts does not improve model performance across a range of questions compared to the control setting where no persona is added. Nevertheless, further analysis suggests that the gender, type, and domain of the persona can all influence the resulting prediction accuracies. We further experimented with a list of persona search strategies and found that, while aggregating results from the best persona for each question significantly improves prediction accuracy, automatically identifying the best persona is challenging, with predictions often performing no better than random selection. Overall, our findings suggest that while adding a persona may lead to performance gains in certain settings, the effect of each persona can be largely random. Code and data are available at https://github.com/Jiaxin-Pei/Prompting-with-Social-Roles.

Figures (24)

• Figure 1: Our overall research question: does adding personas in prompts affect LLMs' performance?
• Figure 2: The first and last 10 personas ranked by their effect sizes on the models' performance change compared with the control setting. Overall, none of the personas lead to statistically better model performance.
• Figure 3: Audience-specific prompts are significantly better than speaker-specific prompts with small effect sizes.
• Figure 4: Most of the personas have no or negative impact on LLM's performance.
• Figure 5: LLMs' sensitivity to persona does not scale with model sizes