Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Realistic Synthetic User-Generated Content: A Scaffolding Approach to Generating Online Discussions

Krisztian Balog, John Palowitch, Barbara Ikica, Filip Radlinski, Hamidreza Alvari, Mehdi Manshadi

TL;DR

The paper studies generating realistic synthetic user-generated content (UGC) by focusing on online discussion threads. It formalizes threads as sequences of posts with parent relationships, and introduces a scaffolded, multi-step pipeline that decouples thread structure (scaffolds) from content generation, using topic extraction and sampling to guide synthetic generation. A novel realism measure and a comprehensive evaluation suite—including topic, structural, content, and realism metrics—are proposed, and the framework is evaluated on Reddit and Wikipedia Talk Pages using PaLM 2 with both few-shot and fine-tuned scaffold strategies. Results show scaffolded generation substantially improves validity and structural realism over baseline end-to-end generation, with strong cross-community generalization and scalable costs, while highlighting areas for future work in prompt design, model fine-tuning, and privacy considerations.

Abstract

The emergence of synthetic data represents a pivotal shift in modern machine learning, offering a solution to satisfy the need for large volumes of data in domains where real data is scarce, highly private, or difficult to obtain. We investigate the feasibility of creating realistic, large-scale synthetic datasets of user-generated content, noting that such content is increasingly prevalent and a source of frequently sought information. Large language models (LLMs) offer a starting point for generating synthetic social media discussion threads, due to their ability to produce diverse responses that typify online interactions. However, as we demonstrate, straightforward application of LLMs yields limited success in capturing the complex structure of online discussions, and standard prompting mechanisms lack sufficient control. We therefore propose a multi-step generation process, predicated on the idea of creating compact representations of discussion threads, referred to as scaffolds. Our framework is generic yet adaptable to the unique characteristics of specific social media platforms. We demonstrate its feasibility using data from two distinct online discussion platforms. To address the fundamental challenge of ensuring the representativeness and realism of synthetic data, we propose a portfolio of evaluation measures to compare various instantiations of our framework.

Towards Realistic Synthetic User-Generated Content: A Scaffolding Approach to Generating Online Discussions

TL;DR

The paper studies generating realistic synthetic user-generated content (UGC) by focusing on online discussion threads. It formalizes threads as sequences of posts with parent relationships, and introduces a scaffolded, multi-step pipeline that decouples thread structure (scaffolds) from content generation, using topic extraction and sampling to guide synthetic generation. A novel realism measure and a comprehensive evaluation suite—including topic, structural, content, and realism metrics—are proposed, and the framework is evaluated on Reddit and Wikipedia Talk Pages using PaLM 2 with both few-shot and fine-tuned scaffold strategies. Results show scaffolded generation substantially improves validity and structural realism over baseline end-to-end generation, with strong cross-community generalization and scalable costs, while highlighting areas for future work in prompt design, model fine-tuning, and privacy considerations.

Abstract

The emergence of synthetic data represents a pivotal shift in modern machine learning, offering a solution to satisfy the need for large volumes of data in domains where real data is scarce, highly private, or difficult to obtain. We investigate the feasibility of creating realistic, large-scale synthetic datasets of user-generated content, noting that such content is increasingly prevalent and a source of frequently sought information. Large language models (LLMs) offer a starting point for generating synthetic social media discussion threads, due to their ability to produce diverse responses that typify online interactions. However, as we demonstrate, straightforward application of LLMs yields limited success in capturing the complex structure of online discussions, and standard prompting mechanisms lack sufficient control. We therefore propose a multi-step generation process, predicated on the idea of creating compact representations of discussion threads, referred to as scaffolds. Our framework is generic yet adaptable to the unique characteristics of specific social media platforms. We demonstrate its feasibility using data from two distinct online discussion platforms. To address the fundamental challenge of ensuring the representativeness and realism of synthetic data, we propose a portfolio of evaluation measures to compare various instantiations of our framework.
Paper Structure (51 sections, 4 equations, 4 figures, 7 tables)

This paper contains 51 sections, 4 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Statement
  4. Scoping Synthetic UGC Generation
  5. Problem Definition
  6. Synthetic Data Generation
  7. Topic Extraction and Sampling
  8. Topic Extraction
  9. Independent Topic Sampling
  10. Conditional Topic Sampling
  11. Baseline Thread Generation
  12. Scaffolded Thread Generation
  13. Creating Example Scaffolds
  14. Few-shot Scaffold Generation
  15. Fine-tuned Scaffold Generation
  16. ...and 36 more sections

Figures (4)

  • Figure 1: Overview of our synthetic data generation framework. Squared rectangles indicate processes, while rounded rectangles with shaded backgrounds signify models. Approaches for thread generation are detailed in Figure \ref{['fig:thread_generation']}.
  • Figure 2: Thread generation approaches. Dashed lines indicate optional input, i.e., the task may be performed either as zero-shot or as few-shot prompting the LLM.
  • Figure 3: Example scaffold.
  • Figure 4: F1 scores across subreddits of the discussion-path realism prompting scheme. The score is computed on a balanced sample of real discussion-paths and "corrupted" discussion paths with some randomly-chosen posts replaced with randomly-chosen posts from the rest of the subreddit.