Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Detecting Ambiguities to Guide Query Rewrite for Robust Conversations in Enterprise AI Assistants

Md Mehrab Tanjim, Xiang Chen, Victor S. Bursztyn, Uttaran Bhattacharya, Tung Mai, Vaishnavi Muppala, Akash Maharaj, Saayan Mitra, Eunyee Koh, Yunyao Li, Ken Russell

TL;DR

The paper tackles ambiguities in multi-turn enterprise AI assistants by introducing an Ambiguity-guided Query Rewrite (AGQR) framework that combines an ambiguity-detection classifier with a selective query-rewrite module. It builds a taxonomy of Pragmatic, Syntactical, and Lexical ambiguities from real logs and leverages a hybrid detection approach that uses both small language models and rule-based signals, along with data augmentation, to improve detection performance. The QR component is model-agnostic and can be powered by LLMs (e.g., GPT-3.5-Turbo, Llama-3.1-70B) or other prompted models, rewriting only ambiguous queries based on chat history to yield unambiguous, query-specific rewrites. The approach is validated with extensive experiments, showing superior ambiguity detection and improved end-to-end robustness, and is deployed in Adobe Experience Platform AI Assistant, reducing downstream errors and latency in production settings.

Abstract

Multi-turn conversations with an Enterprise AI Assistant can be challenging due to conversational dependencies in questions, leading to ambiguities and errors. To address this, we propose an NLU-NLG framework for ambiguity detection and resolution through reformulating query automatically and introduce a new task called "Ambiguity-guided Query Rewrite." To detect ambiguities, we develop a taxonomy based on real user conversational logs and draw insights from it to design rules and extract features for a classifier which yields superior performance in detecting ambiguous queries, outperforming LLM-based baselines. Furthermore, coupling the query rewrite module with our ambiguity detecting classifier shows that this end-to-end framework can effectively mitigate ambiguities without risking unnecessary insertions of unwanted phrases for clear queries, leading to an improvement in the overall performance of the AI Assistant. Due to its significance, this has been deployed in the real world application, namely Adobe Experience Platform AI Assistant.

Detecting Ambiguities to Guide Query Rewrite for Robust Conversations in Enterprise AI Assistants

TL;DR

The paper tackles ambiguities in multi-turn enterprise AI assistants by introducing an Ambiguity-guided Query Rewrite (AGQR) framework that combines an ambiguity-detection classifier with a selective query-rewrite module. It builds a taxonomy of Pragmatic, Syntactical, and Lexical ambiguities from real logs and leverages a hybrid detection approach that uses both small language models and rule-based signals, along with data augmentation, to improve detection performance. The QR component is model-agnostic and can be powered by LLMs (e.g., GPT-3.5-Turbo, Llama-3.1-70B) or other prompted models, rewriting only ambiguous queries based on chat history to yield unambiguous, query-specific rewrites. The approach is validated with extensive experiments, showing superior ambiguity detection and improved end-to-end robustness, and is deployed in Adobe Experience Platform AI Assistant, reducing downstream errors and latency in production settings.

Abstract

Multi-turn conversations with an Enterprise AI Assistant can be challenging due to conversational dependencies in questions, leading to ambiguities and errors. To address this, we propose an NLU-NLG framework for ambiguity detection and resolution through reformulating query automatically and introduce a new task called "Ambiguity-guided Query Rewrite." To detect ambiguities, we develop a taxonomy based on real user conversational logs and draw insights from it to design rules and extract features for a classifier which yields superior performance in detecting ambiguous queries, outperforming LLM-based baselines. Furthermore, coupling the query rewrite module with our ambiguity detecting classifier shows that this end-to-end framework can effectively mitigate ambiguities without risking unnecessary insertions of unwanted phrases for clear queries, leading to an improvement in the overall performance of the AI Assistant. Due to its significance, this has been deployed in the real world application, namely Adobe Experience Platform AI Assistant.

Paper Structure

This paper contains 15 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Approach
  4. Proposed Pipeline
  5. Taxonomy of Ambiguous Queries
  6. Detecting Ambiguities
  7. Pragmatic and Syntactical Ambiguities
  8. Lexical Ambiguities
  9. Query Rewrite
  10. Experimental Setup
  11. Experiments and Results
  12. Conclusion
  13. Model Configurations and Prompt
  14. Lexical Ambiguity Detection with an Example
  15. Data Augmentation

Figures (3)

  • Figure 1: Multi-turn conversations can have dependencies in prior chats, leading to ambiguities and errors (middle). While LLM-based rewriting can resolve some ambiguities, it may also introduce errors through unwanted rephrasing (left). Our proposed NLU-NLG framework, Ambiguity-guided Query Rewrite, rewrites only predicted unclear queries to prevent unnecessary rewrites leading to correct answers (right).
  • Figure 2: Left: Proposed pipeline. Right: Architecture of our proposed ambiguity detection model.
  • Figure 3: The box plots of the features from our training set show a clear difference in distribution between clear and ambiguous queries.