Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Operation Veja: Fixing Fundamental Concepts Missing from Modern Roleplaying Training Paradigms

Yueze Liu, Ajay Nagi Reddy Kumdam, Ronit Kanjilal, Hao Yang, Yichi Zhang

TL;DR

The paper argues that modern roleplaying models fail to capture the dynamic internal value systems that drive believable characters. It introduces the VEJA framework (Values, Experiences, Judgments, Abilities) as a grounded data-curation approach and presents a pilot study contrasting VEJA-guided, human-authored dialogue with a synthetic baseline. Using an LLM-as-judge evaluation, the VEJA data show a clear quality gap, suggesting that conceptually grounded data curation yields deeper, narratively coherent characters. The work highlights the need to shift from surface traits to deliberative identity modeling and outlines future directions for scaling data creation and developing robust evaluation metrics.

Abstract

Modern roleplaying models are increasingly sophisticated, yet they consistently struggle to capture the essence of believable, engaging characters. We argue this failure stems from training paradigms that overlook the dynamic interplay of a character's internal world. Current approaches, including Retrieval-Augmented Generation (RAG), fact-based priming, literature-based learning, and synthetic data generation, exhibit recurring limitations in modeling the deliberative, value-conflicted reasoning that defines human interaction. In this paper, we identify four core concepts essential for character authenticity: Values, Experiences, Judgments, and Abilities (VEJA). We propose the VEJA framework as a new paradigm for data curation that addresses these systemic limitations. To illustrate the qualitative ceiling enabled by our framework, we present a pilot study comparing a manually curated, VEJA-grounded dataset against a state-of-the-art synthetic baseline. Using an LLM-as-judge evaluation, our findings demonstrate a significant quality gap, suggesting that a shift toward conceptually grounded data curation, as embodied by VEJA, is necessary for creating roleplaying agents with genuine depth and narrative continuity. The full dataset is available at https://github.com/HyouinKyoumaIRL/Operation-Veja

Operation Veja: Fixing Fundamental Concepts Missing from Modern Roleplaying Training Paradigms

TL;DR

The paper argues that modern roleplaying models fail to capture the dynamic internal value systems that drive believable characters. It introduces the VEJA framework (Values, Experiences, Judgments, Abilities) as a grounded data-curation approach and presents a pilot study contrasting VEJA-guided, human-authored dialogue with a synthetic baseline. Using an LLM-as-judge evaluation, the VEJA data show a clear quality gap, suggesting that conceptually grounded data curation yields deeper, narratively coherent characters. The work highlights the need to shift from surface traits to deliberative identity modeling and outlines future directions for scaling data creation and developing robust evaluation metrics.

Abstract

Modern roleplaying models are increasingly sophisticated, yet they consistently struggle to capture the essence of believable, engaging characters. We argue this failure stems from training paradigms that overlook the dynamic interplay of a character's internal world. Current approaches, including Retrieval-Augmented Generation (RAG), fact-based priming, literature-based learning, and synthetic data generation, exhibit recurring limitations in modeling the deliberative, value-conflicted reasoning that defines human interaction. In this paper, we identify four core concepts essential for character authenticity: Values, Experiences, Judgments, and Abilities (VEJA). We propose the VEJA framework as a new paradigm for data curation that addresses these systemic limitations. To illustrate the qualitative ceiling enabled by our framework, we present a pilot study comparing a manually curated, VEJA-grounded dataset against a state-of-the-art synthetic baseline. Using an LLM-as-judge evaluation, our findings demonstrate a significant quality gap, suggesting that a shift toward conceptually grounded data curation, as embodied by VEJA, is necessary for creating roleplaying agents with genuine depth and narrative continuity. The full dataset is available at https://github.com/HyouinKyoumaIRL/Operation-Veja
Paper Structure (20 sections, 3 figures)

This paper contains 20 sections, 3 figures.

Table of Contents

  1. Introduction
  2. A Critical Analysis of Current Roleplaying Paradigms
  3. Retrieval-Augmented Generation (RAG): The Scaling Problem
  4. Fact-Based Value Priming: The Decontextualization Problem
  5. Literature-Based Generation: The Curse of Implicit Context and Complexity
  6. Synthetic Data Generation and Distillation: The Chicken-and-Egg Problem
  7. The VEJA Framework
  8. Values (V)
  9. Experiences (E)
  10. Judgments (J)
  11. Abilities (A)
  12. Case Study: Curating a VEJA-Grounded Dataset
  13. Dataset Curation
  14. Evaluation Methodology
  15. Results
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Conceptual cycle illustrating how Values, Experiences, Judgments, and Abilities interact. Experiences shape Values, which together inform Judgments, expressed through Abilities.
  • Figure 2: Experimental pipeline showing the baseline (top) and VEJA (bottom) branches, both feeding into the LLM-as-judge evaluation.
  • Figure 3: LLM-as-Judge Preference in Pairwise A/B Testing (N=100). The VEJA-curated dialogues were significantly preferred over the synthetically generated baseline.