Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Synthetic Clarification and Correction Dialogues about Data-Centric Tasks -- A Teacher-Student Approach

Christian Poelitz, Nick McKenna

TL;DR

The paper proposes a novel teacher-student framework to generate synthetic, solvable curricula of clarifications and corrections for table-based question answering, enabling scalable benchmarking and training from existing data-centric tasks. By ablating information and using a strong teacher to supervise student-generated clarifications or corrections, the method creates multi-turn dialogues that reflect realistic user-AI interactions. Experiments on TaT-QA and WikiTableQuestions show that larger models better leverage corrections than clarifications, while finetuning on the synthetic curriculum consistently improves recall and final accuracy for several models. The work demonstrates the feasibility and value of synthetic, teacher-verified dialogue data for data-centric tasks, while highlighting ongoing challenges in model integration of user feedback and the costs associated with teacher-driven data generation.

Abstract

Real dialogues with AI assistants for solving data-centric tasks often follow dynamic, unpredictable paths due to imperfect information provided by the user or in the data, which must be caught and handled. Developing datasets which capture such user-AI interactions is difficult and time-consuming. In this work, we develop a novel framework for synthetically generating controlled, multi-turn conversations between a user and AI assistant for the task of table-based question answering, which can be generated from an existing dataset with fully specified table QA examples for any target domain. Each conversation aims to solve a table-based reasoning question through collaborative effort, modeling one of two real-world scenarios: (1) an AI-initiated clarification, or (2) a user-initiated correction. Critically, we employ a strong teacher LLM to verify the correctness of our synthetic conversations, ensuring high quality. We demonstrate synthetic datasets generated from TAT-QA and WikiTableQuestions as benchmarks of frontier LLMs. We find that even larger models struggle to effectively issuing clarification questions and accurately integrate user feedback for corrections.

Synthetic Clarification and Correction Dialogues about Data-Centric Tasks -- A Teacher-Student Approach

TL;DR

The paper proposes a novel teacher-student framework to generate synthetic, solvable curricula of clarifications and corrections for table-based question answering, enabling scalable benchmarking and training from existing data-centric tasks. By ablating information and using a strong teacher to supervise student-generated clarifications or corrections, the method creates multi-turn dialogues that reflect realistic user-AI interactions. Experiments on TaT-QA and WikiTableQuestions show that larger models better leverage corrections than clarifications, while finetuning on the synthetic curriculum consistently improves recall and final accuracy for several models. The work demonstrates the feasibility and value of synthetic, teacher-verified dialogue data for data-centric tasks, while highlighting ongoing challenges in model integration of user feedback and the costs associated with teacher-driven data generation.

Abstract

Real dialogues with AI assistants for solving data-centric tasks often follow dynamic, unpredictable paths due to imperfect information provided by the user or in the data, which must be caught and handled. Developing datasets which capture such user-AI interactions is difficult and time-consuming. In this work, we develop a novel framework for synthetically generating controlled, multi-turn conversations between a user and AI assistant for the task of table-based question answering, which can be generated from an existing dataset with fully specified table QA examples for any target domain. Each conversation aims to solve a table-based reasoning question through collaborative effort, modeling one of two real-world scenarios: (1) an AI-initiated clarification, or (2) a user-initiated correction. Critically, we employ a strong teacher LLM to verify the correctness of our synthetic conversations, ensuring high quality. We demonstrate synthetic datasets generated from TAT-QA and WikiTableQuestions as benchmarks of frontier LLMs. We find that even larger models struggle to effectively issuing clarification questions and accurately integrate user feedback for corrections.

Paper Structure

This paper contains 24 sections, 1 equation, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background and related work
  3. Synthetic data generation
  4. Dialogue systems
  5. Data-centric tasks
  6. Framework
  7. Student-teacher method
  8. Ablation strategies
  9. Clarification
  10. Corrections
  11. Curriculum
  12. Teacher Model Analysis
  13. Teacher Model
  14. Task ablation
  15. User responses
  16. ...and 9 more sections

Figures (5)

  • Figure 1: Illustration of synthetic corrections and clarifications based on table QA tasks from a dataset. Top: Original table QA dataset. Bottom: Two synthetic conversations with underspecified question. Top-bottom: User-initiated correction. Lower-bottom: AI-initiated clarification.
  • Figure 2: Teacher-Student framework illustrating synthetic dialogue generation with clarifications and corrections. Starting with an initial (single turn) table QA task, the teacher model rephrases and removes information such that the student cannot solve it (anymore). Then, the teacher guides the student how to ask clarification questions and how to use the provided user corrections to solve the task.
  • Figure 3: Prompt for the teacher model to judge a students answers for table QA tasks.
  • Figure 4: Prompt for the teacher model to ablate information from the user question.
  • Figure 5: Prompt for the teacher model to ablate information from the table.