Dynamic Context Adaptation for Consistent Role-Playing Agents with Retrieval-Augmented Generations

TL;DR

This work tackles the difficulty of maintaining persona-consistent behavior in retrieval-augmented role-playing agents when queries extend beyond a character’s explicit knowledge. It introduces Amadeus, a training-free framework composed of Adaptive Context-aware Text Splitter (ACTS), Guided Selection (GS), and Attribute Extractor (AE) to adaptively structure persona contexts, infer character attributes, and generate faithful responses. The CharacterRAG dataset provides 15 fictional personas with 976K characters and 450 QA pairs to benchmark RAG-based RPAs, enabling rigorous evaluation of persona fidelity across multiple LLMs. Experimental results show Amadeus improves persona consistency and reduces hallucination compared with several baselines, highlighting the value of adaptive chunking, inference-driven retrieval, and attribute integration for practical RAG-based role-playing. This work offers a resource and approach that can significantly impact the development of robust, believable RPAs in interactive settings.

Abstract

Building role-playing agents (RPAs) that faithfully emulate specific characters remains challenging because collecting character-specific utterances and continually updating model parameters are resource-intensive, making retrieval-augmented generation (RAG) a practical necessity. However, despite the importance of RAG, there has been little research on RAG-based RPAs. For example, we empirically find that when a persona lacks knowledge relevant to a given query, RAG-based RPAs are prone to hallucination, making it challenging to generate accurate responses. In this paper, we propose Amadeus, a training-free framework that can significantly enhance persona consistency even when responding to questions that lie beyond a character's knowledge. In addition, to underpin the development and rigorous evaluation of RAG-based RPAs, we manually construct CharacterRAG, a role-playing dataset that consists of persona documents for 15 distinct fictional characters totaling 976K written characters, and 450 question-answer pairs. We find that our proposed method effectively models not only the knowledge possessed by characters, but also various attributes such as personality.

Figures (8)

• Figure 1: Comparison of Responses between Naive RAG and Our Method. We observe that the existing RAG method tends to excessively utilize chunks that are less relevant to the question when the question is not explicitly answered by the available knowledge.
• Figure 2: An overview of CharacterRAG Dataset. CharacterRAG consists of persona documents for 15 distinct fictional characters totaling 976K written characters, and 450 question–answer pairs.
• Figure 3: Prompt used in Guided Selection (GS). GS returns whether Belief and Value information or Psychological Traits can be inferred from given chunks.
• Figure 4: Prompt ued in the interview-based psychological assessment (MBTI). In the BFI test, responses are rated on a scale from 1 to 5.
• Figure 5: AMADEUS framework. AMADEUS consists of three substage: (i) ACTS splits a persona document to make it suitable for RAG-based role-playing. (ii) To fully leverage the character’s knowledge, GS retrieves chunks from which it can infer the answer to the user query. (iii) AE uses the information derived from the GS results to extract character attributes.