Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Every time I fire a conversational designer, the performance of the dialog system goes down

Giancarlo A. Xompero, Michele Mastromattei, Samir Salman, Cristina Giannone, Andrea Favalli, Raniero Romagnoli, Fabio Massimo Zanzotto

TL;DR

The paper tackles the data efficiency problem in neural task oriented dialogue by injecting explicit domain knowledge through a semi-logical symbolic layer CLINN that sits on top of a Domain Aware Multi-Decoder framework. By representing dialogue states with $S_t=(R_t,B_t,A_t)$ and encoding designer authored Horn clauses as belief and action rules, CLINN can be invoked alone or with DAMD, enabling controlled knowledge infusion. Experiments on the MultiWOZ Restaurant domain show that expert designer rules substantially improve key metrics such as Joint Goal, Slot Accuracy, and Slot F1, especially under data scarcity, and that partially constrained (Free) rules often outperform fully constrained ones. The findings suggest a practical shift toward investing in experienced conversational designers to reduce annotation needs while achieving strong performance, highlighting the value of explicit knowledge integration in neural dialogue systems.

Abstract

Incorporating explicit domain knowledge into neural-based task-oriented dialogue systems is an effective way to reduce the need of large sets of annotated dialogues. In this paper, we investigate how the use of explicit domain knowledge of conversational designers affects the performance of neural-based dialogue systems. To support this investigation, we propose the Conversational-Logic-Injection-in-Neural-Network system (CLINN) where explicit knowledge is coded in semi-logical rules. By using CLINN, we evaluated semi-logical rules produced by a team of differently skilled conversational designers. We experimented with the Restaurant topic of the MultiWOZ dataset. Results show that external knowledge is extremely important for reducing the need of annotated examples for conversational systems. In fact, rules from conversational designers used in CLINN significantly outperform a state-of-the-art neural-based dialogue system.

Every time I fire a conversational designer, the performance of the dialog system goes down

TL;DR

The paper tackles the data efficiency problem in neural task oriented dialogue by injecting explicit domain knowledge through a semi-logical symbolic layer CLINN that sits on top of a Domain Aware Multi-Decoder framework. By representing dialogue states with and encoding designer authored Horn clauses as belief and action rules, CLINN can be invoked alone or with DAMD, enabling controlled knowledge infusion. Experiments on the MultiWOZ Restaurant domain show that expert designer rules substantially improve key metrics such as Joint Goal, Slot Accuracy, and Slot F1, especially under data scarcity, and that partially constrained (Free) rules often outperform fully constrained ones. The findings suggest a practical shift toward investing in experienced conversational designers to reduce annotation needs while achieving strong performance, highlighting the value of explicit knowledge integration in neural dialogue systems.

Abstract

Incorporating explicit domain knowledge into neural-based task-oriented dialogue systems is an effective way to reduce the need of large sets of annotated dialogues. In this paper, we investigate how the use of explicit domain knowledge of conversational designers affects the performance of neural-based dialogue systems. To support this investigation, we propose the Conversational-Logic-Injection-in-Neural-Network system (CLINN) where explicit knowledge is coded in semi-logical rules. By using CLINN, we evaluated semi-logical rules produced by a team of differently skilled conversational designers. We experimented with the Restaurant topic of the MultiWOZ dataset. Results show that external knowledge is extremely important for reducing the need of annotated examples for conversational systems. In fact, rules from conversational designers used in CLINN significantly outperform a state-of-the-art neural-based dialogue system.

Paper Structure

This paper contains 15 sections, 3 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Method and System
  4. Domain Aware Multi-Decoder (DAMD) network
  5. Hand-crafting a Dialogue State Tracker
  6. Dialogue States in a Semi-logical Language
  7. Transition Rules in a Semi-logical Language
  8. Experiments
  9. Experimental Set-Up
  10. General principles and Dialogue corpus
  11. Editing and Evaluating Transition rules in Dialogue State Trackers
  12. Evaluation metrics for Dialogue systems
  13. Configurations and Meta-parameters
  14. Results and Discussion
  15. Conclusions

Figures (2)

  • Figure 1: Injecting External Knowledge with CLINN and Architecture of the Domain Aware Multi-Decoder (DAMD) network
  • Figure 2: Trend of models' performances when increasing the number of dialogue examples used to train DAMD. The plots are showing results of the configuration where CLINN uses the Medium set of the Free Rule type