ECLAIR: Enhanced Clarification for Interactive Responses in an Enterprise AI Assistant
John Murzaku, Zifan Liu, Vaishnavi Muppala, Md Mehrab Tanjim, Xiang Chen, Yunyao Li
TL;DR
Problem: LLMs struggle with ambiguity in enterprise-scale interactions due to limited context and domain knowledge. Approach: ECLAIR is a modular, multi-agent framework that gathers ambiguity signals from domain-specific agents, aggregates them into prompts, and uses an LLM to decide if clarification is needed, generating a two-option clarification prompt when appropriate. Dataset and findings: evaluated on real Adobe Experience Platform (AEP) AI Assistant queries, with $P$, $R$, and $F_1$ metrics; average $F_1$ improves from baseline ~0.520 to ~0.657, driven by improved precision in non-clarification cases and stronger non-clarification recall. Significance: enables context-rich, low-latency disambiguation for enterprise assistants, reducing unnecessary clarifications and enhancing user experience.
Abstract
Large language models (LLMs) have shown remarkable progress in understanding and generating natural language across various applications. However, they often struggle with resolving ambiguities in real-world, enterprise-level interactions, where context and domain-specific knowledge play a crucial role. In this demonstration, we introduce ECLAIR (Enhanced CLArification for Interactive Responses), a multi-agent framework for interactive disambiguation. ECLAIR enhances ambiguous user query clarification through an interactive process where custom agents are defined, ambiguity reasoning is conducted by the agents, clarification questions are generated, and user feedback is leveraged to refine the final response. When tested on real-world customer data, ECLAIR demonstrates significant improvements in clarification question generation compared to standard few-shot methods.