Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Your Next Token Prediction: A Multilingual Benchmark for Personalized Response Generation

Shiyao Ding, Takayuki Ito

TL;DR

The paper introduces Your Next Token Prediction (YNTP), a multilingual benchmark for personalized response generation that captures token-level decisions across daily conversations with MBTI-grounded NPCs. It formalizes the task as predicting a user’s next-token responses conditioned on context and user background, and it evaluates prompt-based and parameter-efficient tuning methods on English, Chinese, and Japanese data gathered from a five-day multi-NPC dialogue system. The study provides six evaluation metrics spanning semantic fidelity and stylistic consistency, and reports that few-shot prompting, especially when augmented with MBTI inference, yields substantial gains, with LoRA offering niche advantages in style and history tracking. By releasing a 100-user multilingual dataset and establishing baseline results, the work advances user-aligned language modeling and highlights practical trade-offs between external prompting and internal personalization for real-world personalized AI assistants.

Abstract

Large language models (LLMs) excel at general next-token prediction but still struggle to generate responses that reflect how individuals truly communicate, such as replying to emails or social messages in their own style. However, real SNS or email histories are difficult to collect due to privacy concerns. To address this, we propose the task of "Your Next Token Prediction (YNTP)", which models a user's precise word choices through controlled human-agent conversations. We build a multilingual benchmark of 100 dialogue sessions across English, Japanese, and Chinese, where users interact for five days with psychologically grounded NPCs based on MBTI dimensions. This setup captures natural, daily-life communication patterns and enables analysis of users' internal models. We evaluate prompt-based and fine-tuning-based personalization methods, establishing the first benchmark for YNTP and a foundation for user-aligned language modeling. The dataset is available at: https://github.com/AnonymousHub4Submissions/your-next-token-prediction-dataset-100

Your Next Token Prediction: A Multilingual Benchmark for Personalized Response Generation

TL;DR

The paper introduces Your Next Token Prediction (YNTP), a multilingual benchmark for personalized response generation that captures token-level decisions across daily conversations with MBTI-grounded NPCs. It formalizes the task as predicting a user’s next-token responses conditioned on context and user background, and it evaluates prompt-based and parameter-efficient tuning methods on English, Chinese, and Japanese data gathered from a five-day multi-NPC dialogue system. The study provides six evaluation metrics spanning semantic fidelity and stylistic consistency, and reports that few-shot prompting, especially when augmented with MBTI inference, yields substantial gains, with LoRA offering niche advantages in style and history tracking. By releasing a 100-user multilingual dataset and establishing baseline results, the work advances user-aligned language modeling and highlights practical trade-offs between external prompting and internal personalization for real-world personalized AI assistants.

Abstract

Large language models (LLMs) excel at general next-token prediction but still struggle to generate responses that reflect how individuals truly communicate, such as replying to emails or social messages in their own style. However, real SNS or email histories are difficult to collect due to privacy concerns. To address this, we propose the task of "Your Next Token Prediction (YNTP)", which models a user's precise word choices through controlled human-agent conversations. We build a multilingual benchmark of 100 dialogue sessions across English, Japanese, and Chinese, where users interact for five days with psychologically grounded NPCs based on MBTI dimensions. This setup captures natural, daily-life communication patterns and enables analysis of users' internal models. We evaluate prompt-based and fine-tuning-based personalization methods, establishing the first benchmark for YNTP and a foundation for user-aligned language modeling. The dataset is available at: https://github.com/AnonymousHub4Submissions/your-next-token-prediction-dataset-100
Paper Structure (31 sections, 7 equations, 4 figures, 3 tables)

This paper contains 31 sections, 7 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Personalized Alignment Datasets
  4. Personalized Alignment Methods
  5. (1) External Personalization (Without Modifying Parameters).
  6. (2) Internal Personalization (With Modifying Parameters).
  7. Problem Definition: Your Next Token Prediction
  8. Task Formulation.
  9. Learning Objective.
  10. Interpretation.
  11. LLM-Driven Multi-NPC Dialogue System for Benchmark Construction
  12. Personality Modeling via MBTI.
  13. Framework Overview.
  14. Example Scenario.
  15. Data Collection for Benchmarking.
  16. ...and 16 more sections

Figures (4)

  • Figure 1: FSM-based control flow of the LLM-driven multi-NPC dialogue system. Each state corresponds to a query probing one MBTI dimension. User responses are evaluated by a check function that determines transition or reflection.
  • Figure 2: Dialogue interface (shared-house environment and status panel) visualizing user–NPC interactions across multiple personality dimensions.
  • Figure 3: Per-language performance heatmaps (compact layout). Rows are methods with base models; columns are metrics M1--M6 (M1 lower is better; others higher is better). Abbrev.: PE (0)=zero-shot, PE (F)=few-shot, PE (F+MBTI)=few-shot+MBTI, LoRA=PEFT (LoRA).
  • Figure 4: Win/Tie/Lose across metrics (M1--M6) over all languages and base models. Each panel summarizes the percentage of configurations a method wins/ties/loses on that metric. Few-shot methods dominate zero-shot; adding MBTI nudges wins on M1/M3/M5; LoRA is competitive on style (M5) and history (M6).