Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dialog Flow Induction for Constrainable LLM-Based Chatbots

Stuti Agrawal, Nishi Uppuluri, Pranav Pillai, Revanth Gangi Reddy, Zoey Li, Gokhan Tur, Dilek Hakkani-Tur, Heng Ji

TL;DR

The paper tackles the problem of constraining LLM-based chatbots to domain-specific conversations by automatically inducing domain-aware dialog flows. It introduces two unsupervised variants: intrinsic flows, built from GPT-4 knowledge with self-refinement, and data-guided flows, grounded in actual domain dialogs and merged with the intrinsic flow to maximize coverage. Through human and automatic evaluations across multiple task-oriented domains, the data-guided flow demonstrates improved domain coverage while preserving coherence and conclusiveness, illustrating the value of grounding flows in real conversations. The work provides a practical blueprint for integrating flow-controlled schemas into LLM chatbots and discusses limitations and avenues for future improvement in broader, open-ended domains.

Abstract

LLM-driven dialog systems are used in a diverse set of applications, ranging from healthcare to customer service. However, given their generalization capability, it is difficult to ensure that these chatbots stay within the boundaries of the specialized domains, potentially resulting in inaccurate information and irrelevant responses. This paper introduces an unsupervised approach for automatically inducing domain-specific dialog flows that can be used to constrain LLM-based chatbots. We introduce two variants of dialog flow based on the availability of in-domain conversation instances. Through human and automatic evaluation over various dialog domains, we demonstrate that our high-quality data-guided dialog flows achieve better domain coverage, thereby overcoming the need for extensive manual crafting of such flows.

Dialog Flow Induction for Constrainable LLM-Based Chatbots

TL;DR

The paper tackles the problem of constraining LLM-based chatbots to domain-specific conversations by automatically inducing domain-aware dialog flows. It introduces two unsupervised variants: intrinsic flows, built from GPT-4 knowledge with self-refinement, and data-guided flows, grounded in actual domain dialogs and merged with the intrinsic flow to maximize coverage. Through human and automatic evaluations across multiple task-oriented domains, the data-guided flow demonstrates improved domain coverage while preserving coherence and conclusiveness, illustrating the value of grounding flows in real conversations. The work provides a practical blueprint for integrating flow-controlled schemas into LLM chatbots and discusses limitations and avenues for future improvement in broader, open-ended domains.

Abstract

LLM-driven dialog systems are used in a diverse set of applications, ranging from healthcare to customer service. However, given their generalization capability, it is difficult to ensure that these chatbots stay within the boundaries of the specialized domains, potentially resulting in inaccurate information and irrelevant responses. This paper introduces an unsupervised approach for automatically inducing domain-specific dialog flows that can be used to constrain LLM-based chatbots. We introduce two variants of dialog flow based on the availability of in-domain conversation instances. Through human and automatic evaluation over various dialog domains, we demonstrate that our high-quality data-guided dialog flows achieve better domain coverage, thereby overcoming the need for extensive manual crafting of such flows.
Paper Structure (16 sections, 8 figures, 4 tables)

This paper contains 16 sections, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Dialog Flow Induction
  3. Intrinsic Dialog Flow
  4. Initial Flow Generation:
  5. Flow Feedback and Updates:
  6. Flow Finalization:
  7. Data-Guided Dialog Flow
  8. Identifying Representative Examples:
  9. Flow Generation:
  10. Merging with Intrinsic Flow:
  11. Experiments
  12. Datasets
  13. Human Evaluation of Flow Quality
  14. Automatic Evaluation of Flow Coverage
  15. Conclusion and Future Work
  16. ...and 1 more sections

Figures (8)

  • Figure 1: Figure demonstrating how automatically induced domain-specific dialog flows can be used to constrain chatbots to produce domain-focused responses.
  • Figure 2: Figure showing the process for intrinsic flow induction. An initial flow is first generation which is further refined with feedback, update, and clean-up stages. Detailed prompts for each stage are provided in the appendix.
  • Figure 3: Figure showing the methodology for inducing dialog flows using a data-guided approach. Representative examples from the domain conversation instances are used to condition the GPT-4 prompts.
  • Figure 4: Figure showing prompts for intrinsic and data-guided dialog flow generation.
  • Figure 5: Data-driven (a) and Intrinsic (b) flows for the movie listings domain from MetaLWoz.
  • ...and 3 more figures