Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Synthesizing Conversations from Unlabeled Documents using Automatic Response Segmentation

Fanyou Wu, Weijie Xu, Chandan K. Reddy, Srinivasan H. Sengamedu

TL;DR

This paper tackles the scarcity of high-quality ConvQA training data by generating synthetic, high-fidelity dialogues from unlabeled documents. It introduces SynCARS, a dialog synthesis framework that performs automatic response segmentation using an answer segmentation token, enabling multi-sentence answers and more coherent questions. Through human and GPT-4 evaluations, SynCARS-derived data surpasses WikiDialog in answer quality and question specificity, and when used to pre-train a dual-encoder retriever, yields superior open-domain conversational retrieval performance on OR-QuAC benchmarks. The work demonstrates that carefully segmented, inpainted synthetic data can significantly enhance document-based ConvQA systems while using relatively smaller models and modest compute compared to prior approaches.

Abstract

In this study, we tackle the challenge of inadequate and costly training data that has hindered the development of conversational question answering (ConvQA) systems. Enterprises have a large corpus of diverse internal documents. Instead of relying on a searching engine, a more compelling approach for people to comprehend these documents is to create a dialogue system. In this paper, we propose a robust dialog synthesising method. We learn the segmentation of data for the dialog task instead of using segmenting at sentence boundaries. The synthetic dataset generated by our proposed method achieves superior quality when compared to WikiDialog, as assessed through machine and human evaluations. By employing our inpainted data for ConvQA retrieval system pre-training, we observed a notable improvement in performance across OR-QuAC benchmarks.

Synthesizing Conversations from Unlabeled Documents using Automatic Response Segmentation

TL;DR

This paper tackles the scarcity of high-quality ConvQA training data by generating synthetic, high-fidelity dialogues from unlabeled documents. It introduces SynCARS, a dialog synthesis framework that performs automatic response segmentation using an answer segmentation token, enabling multi-sentence answers and more coherent questions. Through human and GPT-4 evaluations, SynCARS-derived data surpasses WikiDialog in answer quality and question specificity, and when used to pre-train a dual-encoder retriever, yields superior open-domain conversational retrieval performance on OR-QuAC benchmarks. The work demonstrates that carefully segmented, inpainted synthetic data can significantly enhance document-based ConvQA systems while using relatively smaller models and modest compute compared to prior approaches.

Abstract

In this study, we tackle the challenge of inadequate and costly training data that has hindered the development of conversational question answering (ConvQA) systems. Enterprises have a large corpus of diverse internal documents. Instead of relying on a searching engine, a more compelling approach for people to comprehend these documents is to create a dialogue system. In this paper, we propose a robust dialog synthesising method. We learn the segmentation of data for the dialog task instead of using segmenting at sentence boundaries. The synthetic dataset generated by our proposed method achieves superior quality when compared to WikiDialog, as assessed through machine and human evaluations. By employing our inpainted data for ConvQA retrieval system pre-training, we observed a notable improvement in performance across OR-QuAC benchmarks.
Paper Structure (20 sections, 7 equations, 6 figures, 7 tables)

This paper contains 20 sections, 7 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. SynCARS
  4. Notations
  5. Answer Segmentation
  6. Training
  7. Inference
  8. Experimental Setup
  9. Datasets
  10. Models
  11. Evaluation
  12. Quality Comparison
  13. Question Types
  14. GPT-4 vs Human Evaluation
  15. Application to Conversational Retrieval
  16. ...and 5 more sections

Figures (6)

  • Figure 1: An example dialog from WikiDialog (WD) and ours (RQ). The blue and yellow boxes in the dialog contain the questions generated by our approach, while the green boxes contain the corresponding answers. WD asks a question starting with "Are there any other .." which is not useful to train a question answering system. Besides, some of the answers can be combined such as the last two on the left side. In contrast, our method fixed those problems.
  • Figure 1: Comparison examples #1. Here we can find the ability of our proposed method to perform segmentation of the sentences. This link (https://en.wikipedia.org/wiki/Grounding_in_communication) gives the raw Wikipedia web page.
  • Figure 2: An illustration of preparing the training dataset, considering a training instance with a question and two sentences as answers. Here, grey boxes represent the extra_ids tokens for T5.
  • Figure 2: Comparison example #2. Here we can find a relatively better question generation compared to WikiDialog. This link (https://en.wikipedia.org/wiki/2NE1) gives the raw Wikipedia web page.
  • Figure 3: A flowchart illustrating the use of a trained model to convert a document into a dialogue format where, in each iteration, at most $N$ sentences are processed, and only the first generated question is retained.
  • ...and 1 more figures