Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Thinking Before Speaking: A Role-playing Model with Mindset

Baohua Zhang, Yongyi Huang, Wenyao Cui, Huaping Zhang

TL;DR

The paper tackles the difficulty of authentic role-playing by LLMs, where models often lose track of a character’s knowledge and reasoning. It introduces Thinking Before Speaking (TBS), a data-augmented, mindset-aware training approach that uses role profiles, scene-based dialogues, and negative samples to teach refusal for out-of-scope queries, all fine-tuned with LoRA. The framework combines multiple data streams (Role Profile, Dialogues, Hallucination Knowledge) and evaluates with a richer set of metrics, demonstrating improved tone, knowledge adherence, and character thinking. Experimental results on a sizable, multilingual dataset of 152 roles show consistent gains over strong baselines, with ablation analyses confirming the value of thinking, foresight knowledge, and targeted prompts for robust role-playing.

Abstract

Role-playing is an easy task for Large Language Models (LLMs), as they are skilled at simulating human behaviors. Many current studies have enabled LLMs to generate responses in the tone of a specific role by fine-tuning the models or using specialized prompts. However, it is typically easy to recognize when a role is being played by LLMs. These models tend to perform poorly when confronted with knowledge that the assumed role does not possess, or a question that requires the specific experience or logic of the role to answer. To address this problem and make LLMs act more like real roles, we propose a Thinking Before Speaking (TBS) model in this paper. Unlike other studies, we first extend the data based on the character's real-life scenarios and the historical dialogue, supplementing each pair of dialogue with the character's mindset. Then we add few data points that include elements beyond the role's knowledge, and fine-tune the LLMs. This approach can help LLMs adopt the role's thought process and logic, avoiding responses that fall outside the role's knowledge base. We have also prepared a dataset and evaluation metrics to test these capabilities. Experimental results show that our TBS model can better emulate a role in terms of tone, knowledge, and mindset.

Thinking Before Speaking: A Role-playing Model with Mindset

TL;DR

The paper tackles the difficulty of authentic role-playing by LLMs, where models often lose track of a character’s knowledge and reasoning. It introduces Thinking Before Speaking (TBS), a data-augmented, mindset-aware training approach that uses role profiles, scene-based dialogues, and negative samples to teach refusal for out-of-scope queries, all fine-tuned with LoRA. The framework combines multiple data streams (Role Profile, Dialogues, Hallucination Knowledge) and evaluates with a richer set of metrics, demonstrating improved tone, knowledge adherence, and character thinking. Experimental results on a sizable, multilingual dataset of 152 roles show consistent gains over strong baselines, with ablation analyses confirming the value of thinking, foresight knowledge, and targeted prompts for robust role-playing.

Abstract

Role-playing is an easy task for Large Language Models (LLMs), as they are skilled at simulating human behaviors. Many current studies have enabled LLMs to generate responses in the tone of a specific role by fine-tuning the models or using specialized prompts. However, it is typically easy to recognize when a role is being played by LLMs. These models tend to perform poorly when confronted with knowledge that the assumed role does not possess, or a question that requires the specific experience or logic of the role to answer. To address this problem and make LLMs act more like real roles, we propose a Thinking Before Speaking (TBS) model in this paper. Unlike other studies, we first extend the data based on the character's real-life scenarios and the historical dialogue, supplementing each pair of dialogue with the character's mindset. Then we add few data points that include elements beyond the role's knowledge, and fine-tune the LLMs. This approach can help LLMs adopt the role's thought process and logic, avoiding responses that fall outside the role's knowledge base. We have also prepared a dataset and evaluation metrics to test these capabilities. Experimental results show that our TBS model can better emulate a role in terms of tone, knowledge, and mindset.
Paper Structure (16 sections, 2 figures, 19 tables)

This paper contains 16 sections, 2 figures, 19 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Model
  4. Role Profile
  5. Dialogues
  6. Hallucination Knowledge
  7. Fine-tune with Lora
  8. Experiment
  9. Dataset
  10. Metrics
  11. Baseline
  12. Settings
  13. Comparison Experiment
  14. Ablation Experiment
  15. Conclusion
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Overview of the TBS model. We organize the character data into a special prompt and fine-tune the LLMs using LoRA. We input the role profile summary into the prompts, followed by the dialogue pairs, as shown in Table \ref{['example_train_prompt']}.
  • Figure 2: Overview of the data construction.