Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rehearse With User: Personalized Opinion Summarization via Role-Playing based on Large Language Models

Yanyue Zhang, Yulan He, Deyu Zhou

TL;DR

The paper tackles personalized opinion summarization for multi-document product reviews by introducing Rehearsal, a multi-agent framework in which an LLM-based summary agent produces a general summary, a user agent engages in role-playing with supervision to capture individual interests, and a retrieval-augmented rewriting step tailors the final personalized summary. It formalizes role-playing with practice and evaluation against four consistency metrics, and employs retrieval-augmented rewriting to prune irrelevant content and incorporate user-relevant information. The authors curate PerSum, a specialized dataset built on Amazon reviews, and demonstrate that Rehearsal improves personalization alignment across multiple base models, though they note limitations in supervision quality and evaluation metrics. The work advances personalized summarization by combining role-playing, supervision, and retrieval augmentation to better reflect reader interests in summaries, with implications for user-centric recommendation and evaluation in large-language-model-driven systems.

Abstract

Personalized opinion summarization is crucial as it considers individual user interests while generating product summaries. Recent studies show that although large language models demonstrate powerful text summarization and evaluation capabilities without the need for training data, they face difficulties in personalized tasks involving long texts. To address this, \textbf{Rehearsal}, a personalized opinion summarization framework via LLMs-based role-playing is proposed. Having the model act as the user, the model can better understand the user's personalized needs. Additionally, a role-playing supervisor and practice process are introduced to improve the role-playing ability of the LLMs, leading to a better expression of user needs. Furthermore, through suggestions from virtual users, the summary generation is intervened, ensuring that the generated summary includes information of interest to the user, thus achieving personalized summary generation. Experiment results demonstrate that our method can effectively improve the level of personalization in large model-generated summaries.

Rehearse With User: Personalized Opinion Summarization via Role-Playing based on Large Language Models

TL;DR

The paper tackles personalized opinion summarization for multi-document product reviews by introducing Rehearsal, a multi-agent framework in which an LLM-based summary agent produces a general summary, a user agent engages in role-playing with supervision to capture individual interests, and a retrieval-augmented rewriting step tailors the final personalized summary. It formalizes role-playing with practice and evaluation against four consistency metrics, and employs retrieval-augmented rewriting to prune irrelevant content and incorporate user-relevant information. The authors curate PerSum, a specialized dataset built on Amazon reviews, and demonstrate that Rehearsal improves personalization alignment across multiple base models, though they note limitations in supervision quality and evaluation metrics. The work advances personalized summarization by combining role-playing, supervision, and retrieval augmentation to better reflect reader interests in summaries, with implications for user-centric recommendation and evaluation in large-language-model-driven systems.

Abstract

Personalized opinion summarization is crucial as it considers individual user interests while generating product summaries. Recent studies show that although large language models demonstrate powerful text summarization and evaluation capabilities without the need for training data, they face difficulties in personalized tasks involving long texts. To address this, \textbf{Rehearsal}, a personalized opinion summarization framework via LLMs-based role-playing is proposed. Having the model act as the user, the model can better understand the user's personalized needs. Additionally, a role-playing supervisor and practice process are introduced to improve the role-playing ability of the LLMs, leading to a better expression of user needs. Furthermore, through suggestions from virtual users, the summary generation is intervened, ensuring that the generated summary includes information of interest to the user, thus achieving personalized summary generation. Experiment results demonstrate that our method can effectively improve the level of personalization in large model-generated summaries.

Paper Structure

This paper contains 26 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Personalized Opinion Summarization
  4. Role playing and Multi-agent based on LLMs
  5. LLMs Personalization
  6. Methodology
  7. Overview
  8. General Summarization
  9. Role-Playing Suggestions
  10. User Agent Design
  11. Role-Playing Supervisor Design
  12. Retrieval-augmented Personalized Rewriting
  13. Experiments
  14. Datasets
  15. Evaluation Metrics and Baselines
  16. ...and 11 more sections

Figures (3)

  • Figure 1: The difference between previous work and our work, Rehearsal. The summarization system primarily inputs the product review set and is enhanced via the retriever. The user system primarily inputs the user review set and is enhanced using a supervisor.
  • Figure 2: The example (up) and the execution process (below) of Rehearsal. The example includes the output of three steps. Different colors represent different aspects of the product information. The LLMs-based pseudo-user suggests the summary agent add comfort-related (blue) information and reduce price-related (red) information.
  • Figure 3: Relationship between pass rate and iteration count for different models, with and without ICL in user role-playing.