RoleBreak: Character Hallucination as a Jailbreak Attack in Role-Playing Systems

TL;DR

This work introduces RoleBreak, a framework that treats character hallucination in LLM-based role-playing systems as a jailbreaking attack driven by query sparsity and role-query conflict. It provides RoleBreakEval, a dataset to evaluate how well models resist such attacks, and demonstrates that many strong models remain vulnerable even with enhancements. To counteract these vulnerabilities, the authors propose Narrator Mode, which augments context with global outlines and local plots to improve query generalization and resolve role-query conflicts, outperforming traditional refusal-based strategies in reducing hallucinations and enhancing narrative coherence. The findings underscore the limitations of rejection-based defenses and highlight narrative-context augmentation as a promising direction for robust, immersive role-playing experiences. The work also sets a foundation for future exploration of dynamic narrative generation and more nuanced character management in diverse roles.

Abstract

Role-playing systems powered by large language models (LLMs) have become increasingly influential in emotional communication applications. However, these systems are susceptible to character hallucinations, where the model deviates from predefined character roles and generates responses that are inconsistent with the intended persona. This paper presents the first systematic analysis of character hallucination from an attack perspective, introducing the RoleBreak framework. Our framework identifies two core mechanisms-query sparsity and role-query conflict-as key factors driving character hallucination. Leveraging these insights, we construct a novel dataset, RoleBreakEval, to evaluate existing hallucination mitigation techniques. Our experiments reveal that even enhanced models trained to minimize hallucination remain vulnerable to attacks. To address these vulnerabilities, we propose a novel defence strategy, the Narrator Mode, which generates supplemental context through narration to mitigate role-query conflicts and improve query generalization. Experimental results demonstrate that Narrator Mode significantly outperforms traditional refusal-based strategies by reducing hallucinations, enhancing fidelity to character roles and queries, and improving overall narrative coherence.

Figures (2)

• Figure 1: The overview structure of the proposed method. The left side of the attack section illustrates four underlying causes of character hallucinations. The right side of the defence section presents an example of our proposed defence strategy, the Narration Mode, based on GPT-3.5. This method effectively portrays Beethoven's complex relationship with his father, highlighting its ability to ensure deeper narrative coherence, enhanced interactivity, and mitigation of character hallucinations.
• Figure 2: The results of the ablation experiments based on GPT-3.5. The NM is the abbreviation for Narrator Mode. The value of Story Coherence is standardized by dividing it by 5.