BlendX: Complex Multi-Intent Detection with Blended Patterns

TL;DR

This work presents BlendX, a suite of refined datasets featuring more diverse patterns than their predecessors, elevating both its complexity and diversity, and introduces three novel metrics that assess the statistical properties of an utterance related to word count, conjunction use, and pronoun usage.

Abstract

Task-oriented dialogue (TOD) systems are commonly designed with the presumption that each utterance represents a single intent. However, this assumption may not accurately reflect real-world situations, where users frequently express multiple intents within a single utterance. While there is an emerging interest in multi-intent detection (MID), existing in-domain datasets such as MixATIS and MixSNIPS have limitations in their formulation. To address these issues, we present BlendX, a suite of refined datasets featuring more diverse patterns than their predecessors, elevating both its complexity and diversity. For dataset construction, we utilize both rule-based heuristics as well as a generative tool -- OpenAI's ChatGPT -- which is augmented with a similarity-driven strategy for utterance selection. To ensure the quality of the proposed datasets, we also introduce three novel metrics that assess the statistical properties of an utterance related to word count, conjunction use, and pronoun usage. Extensive experiments on BlendX reveal that state-of-the-art MID models struggle with the challenges posed by the new datasets, highlighting the need to reexamine the current state of the MID field. The dataset is available at https://github.com/HYU-NLP/BlendX.

Figures (6)

• Figure 1: An example that underscores the distinct features of MixX and BlendX. In contrast to MixX, which relies on simple concatenations, BlendX steps beyond by simulating more realistic and complex cases often found in real-world conversations.
• Figure 2: An overview of the BlendX construction framework. Initially, we preprocess four source datasets: ATIS, Banking77, CLINC150, and SNIPS. We then select single-intent utterances from these datasets. These utterances are combined using both Manual and Generative approaches. It is important to note that utterances are kept separate and not mixed across datasets. Following the merging process, all resultant datasets are compiled to form BlendX. We particularly highlight non-trivial combinations, such as omissions, which are indicated within the blue rounded box on the rightmost side of the framework. Finally, BlendX is evaluated using three methods: custom metrics, baseline evaluation, and visualization.
• Figure 3: Illustration of the complexity (Left) and methodology (Right) aspects of concatenation. Each approach triggers a distinct part of the possible variations (Middle) arising in the process of concatenation.
• Figure 4: Prompt design for the Generative Approach. The demonstration section showcases $N$ ($=$ 3) examples of combining $k$ ($=$ 2 or 3) utterances, featuring both a successful and a failed case. The [query] lists the sentences to be merged. ChatGPT performs the merging process by filling in the [answer] part. Blue comments are for illustrative purposes only and are not part of the actual prompt.
• Figure 5: Prompt design for solving MID with in-context learning. We employ the few-shot setting where $k$ multi-intent utterances are provided, each associated with up to three intents. The [Answer] part is filled in to predict the intent of [Query]. Appendix \ref{['subsec:prompt_evaluate']} presents detailed illustrations.