Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PREFINE: Personalized Story Generation via Simulated User Critics and User-Specific Rubric Generation

Kentaro Ueda, Takehiro Takayanagi

TL;DR

Personalizing story generation remains challenging due to user data and privacy concerns. PREFINE extends the Critique-and-Refine paradigm with a pseudo-user agent and user-specific rubrics to personalize outputs without parameter updates or explicit feedback, learning from user history to guide critiques and refinements. Evaluations on PerDOC and PerMPST show improved personalization performance while maintaining general story quality comparable to strong baselines, with ablations confirming the importance of the pseudo-user and rubrics. The framework promises broad applicability to other personalized systems, such as dialogue, education, and recommendations, by leveraging user-specific rubrics and simulated critiques.

Abstract

While recent advances in Large Language Models (LLMs) have improved the quality of creative text generation, significant challenges remain in producing personalized stories that reflect individual user preferences. Conventional approaches rely on explicit feedback or fine-tuning, which presents practical issues regarding user burden, data collection, computational costs, and privacy. In this work, we propose PREFINE (Persona-and-Rubric Guided Critique-and-Refine), a novel framework that extends the Critique-and-Refine paradigm to personalization. PREFINE constructs a pseudo-user agent from a user's interaction history and generates user-specific rubrics (evaluation criteria). By having this agent critique and refine outputs on the user's behalf based on these tailored rubrics, our method achieves personalized generation without requiring parameter updates or direct user feedback. We conducted a comprehensive evaluation on the PerDOC and PerMPST story datasets. We designed three baseline methods and several model variants to verify the contribution of each component of our framework. In automatic evaluations (LLM-as-a-Judge), PREFINE achieved higher win rates and statistically significant scores than the baselines, without compromising general story quality. Analysis of the model variants confirmed that both the pseudo-user agent and the user-specific rubrics are crucial for enhancing personalization performance. Beyond story generation, our approach holds potential for enabling efficient personalization in broader applications, such as dialogue systems, education, and recommendation.

PREFINE: Personalized Story Generation via Simulated User Critics and User-Specific Rubric Generation

TL;DR

Personalizing story generation remains challenging due to user data and privacy concerns. PREFINE extends the Critique-and-Refine paradigm with a pseudo-user agent and user-specific rubrics to personalize outputs without parameter updates or explicit feedback, learning from user history to guide critiques and refinements. Evaluations on PerDOC and PerMPST show improved personalization performance while maintaining general story quality comparable to strong baselines, with ablations confirming the importance of the pseudo-user and rubrics. The framework promises broad applicability to other personalized systems, such as dialogue, education, and recommendations, by leveraging user-specific rubrics and simulated critiques.

Abstract

While recent advances in Large Language Models (LLMs) have improved the quality of creative text generation, significant challenges remain in producing personalized stories that reflect individual user preferences. Conventional approaches rely on explicit feedback or fine-tuning, which presents practical issues regarding user burden, data collection, computational costs, and privacy. In this work, we propose PREFINE (Persona-and-Rubric Guided Critique-and-Refine), a novel framework that extends the Critique-and-Refine paradigm to personalization. PREFINE constructs a pseudo-user agent from a user's interaction history and generates user-specific rubrics (evaluation criteria). By having this agent critique and refine outputs on the user's behalf based on these tailored rubrics, our method achieves personalized generation without requiring parameter updates or direct user feedback. We conducted a comprehensive evaluation on the PerDOC and PerMPST story datasets. We designed three baseline methods and several model variants to verify the contribution of each component of our framework. In automatic evaluations (LLM-as-a-Judge), PREFINE achieved higher win rates and statistically significant scores than the baselines, without compromising general story quality. Analysis of the model variants confirmed that both the pseudo-user agent and the user-specific rubrics are crucial for enhancing personalization performance. Beyond story generation, our approach holds potential for enabling efficient personalization in broader applications, such as dialogue systems, education, and recommendation.
Paper Structure (64 sections, 5 equations, 11 figures, 26 tables)

This paper contains 64 sections, 5 equations, 11 figures, 26 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Personalized Text Generation
  4. Story Generation
  5. LLM-based User Modeling and Simulation
  6. Method
  7. PREFINE
  8. Initial Story Generation
  9. Pseudo-User Agent
  10. User-Specific Rubric Generation
  11. Critique-and-Refine Cycle for Personalization
  12. Critique and Feedback Generation
  13. Story Refinement
  14. Experimental Settings
  15. Datasets
  16. ...and 49 more sections

Figures (11)

  • Figure 1: Overview of PREFINE. PREFINE extends the Critique-and-Refine framework by introducing a pseudo-user agent and a user-specific rubric. This enables the generation of personalized stories for the target user without relying on user feedback or additional model fine-tuning.
  • Figure 2: Relationship between users’ rubric suitability ratings and their preference scores for PREFINE-generated stories.
  • Figure 3: Win rate progression against Prompt-Persona (PP) across refinement iterations.
  • Figure 4: Relationship between token length and evaluation outcomes for EPER outputs. (a) Correlation between token count and evaluation scores on the PerMPST dataset. (b) Win rate comparison on a subset where the token size difference between story pairs is within ±10.
  • Figure 5: Interface of the web application used in the user study. The application enables end-to-end evaluation by collecting user preferences (via synopsis ratings and comments), presenting generated stories for evaluation, and collecting user feedback on personalized user-specific rubrics.
  • ...and 6 more figures