Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PLACES: Prompting Language Models for Social Conversation Synthesis

Maximillian Chen, Alexandros Papangelis, Chenyang Tao, Seokhwan Kim, Andy Rosenbaum, Yang Liu, Zhou Yu, Dilek Hakkani-Tur

TL;DR

PLACES demonstrates a prompting-based framework to synthesize open-domain social conversations from a small set of expert-written demonstrations, achieving dialogue quality comparable to human datasets and extending naturally to triadic, multi-party settings. By evaluating with human judgments and downstream fine-tuning, the work shows synthetic data can match or even support training pipelines traditionally reliant on crowdsourced data. The approach highlights practical benefits for scalable data generation, while acknowledging challenges in controllability, computational cost, and safety. Overall, PLACES offers a viable pathway to expand and diversify conversational datasets, with meaningful implications for multi-party dialogue research and real-world AI assistants.

Abstract

Collecting high quality conversational data can be very expensive for most applications and infeasible for others due to privacy, ethical, or similar concerns. A promising direction to tackle this problem is to generate synthetic dialogues by prompting large language models. In this work, we use a small set of expert-written conversations as in-context examples to synthesize a social conversation dataset using prompting. We perform several thorough evaluations of our synthetic conversations compared to human-collected conversations. This includes various dimensions of conversation quality with human evaluation directly on the synthesized conversations, and interactive human evaluation of chatbots fine-tuned on the synthetically generated dataset. We additionally demonstrate that this prompting approach is generalizable to multi-party conversations, providing potential to create new synthetic data for multi-party tasks. Our synthetic multi-party conversations were rated more favorably across all measured dimensions compared to conversation excerpts sampled from a human-collected multi-party dataset.

PLACES: Prompting Language Models for Social Conversation Synthesis

TL;DR

PLACES demonstrates a prompting-based framework to synthesize open-domain social conversations from a small set of expert-written demonstrations, achieving dialogue quality comparable to human datasets and extending naturally to triadic, multi-party settings. By evaluating with human judgments and downstream fine-tuning, the work shows synthetic data can match or even support training pipelines traditionally reliant on crowdsourced data. The approach highlights practical benefits for scalable data generation, while acknowledging challenges in controllability, computational cost, and safety. Overall, PLACES offers a viable pathway to expand and diversify conversational datasets, with meaningful implications for multi-party dialogue research and real-world AI assistants.

Abstract

Collecting high quality conversational data can be very expensive for most applications and infeasible for others due to privacy, ethical, or similar concerns. A promising direction to tackle this problem is to generate synthetic dialogues by prompting large language models. In this work, we use a small set of expert-written conversations as in-context examples to synthesize a social conversation dataset using prompting. We perform several thorough evaluations of our synthetic conversations compared to human-collected conversations. This includes various dimensions of conversation quality with human evaluation directly on the synthesized conversations, and interactive human evaluation of chatbots fine-tuned on the synthetically generated dataset. We additionally demonstrate that this prompting approach is generalizable to multi-party conversations, providing potential to create new synthetic data for multi-party tasks. Our synthetic multi-party conversations were rated more favorably across all measured dimensions compared to conversation excerpts sampled from a human-collected multi-party dataset.
Paper Structure (29 sections, 4 figures, 26 tables)

This paper contains 29 sections, 4 figures, 26 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Conversation Generation
  4. Writing Conversation Examples
  5. Creating Conversations via Prompting
  6. Synthetic Conversation Evaluation
  7. Evaluation of Conversation Quality
  8. Fine-Tuning with Synthetic Conversations
  9. Triadic and Multi-Party Conversations
  10. Discussion
  11. Considerations for Dyadic Dialogue
  12. Considerations for Multi-Party Dialogue
  13. Error Analysis
  14. Dyadic Conversations
  15. Triadic Conversations
  16. ...and 14 more sections

Figures (4)

  • Figure 1: Pair of dyadic conversation excerpts about hometowns (upper) and pair of triadic conversation excerpts about Ithaca, NY (lower). In both pairings, one conversation is synthetically generated and the other is collected from humans. The answer is in Section \ref{['conversation_evaluation']}.
  • Figure 2: Example of the components of a prompt (left) used by OPT 30B to generate a synthetic conversation about pets (right). Conversations in the prompt are prefixed by recipes. Blue text: topic labels. Red text: seed background information metadata.
  • Figure 3: Distinct-N with $N=2,3,4$ for conversations in DailyDialog, Topical Chat, and our synthetic conversations. Our synthetic conversations have the highest most unique bi-grams and tri-grams, and the second-most unique 4-grams.
  • Figure 4: Linguistic diveristy (Distinct-N) is comparable for each speaker in the synthetic triadic conversation dataset.