Empowering Language Models with Active Inquiry for Deeper Understanding

TL;DR

LaMAI introduces an active-inquiry framework that enables language models to selectively ask clarifying questions to resolve ambiguity in user queries. By combining uncertainty estimation via multiple answer sampling with active learning-driven question selection, LaMAI augments user input and yields significantly more accurate and relevant responses across diverse multi-hop Q&A tasks. The approach demonstrates robustness across different LLMs, including Vicuna-13B, and gains are corroborated by both GPT-4-based and human-in-the-loop evaluations. This interactive, context-enhancing strategy has strong practical potential for making LLMs more reliable in real-world, context-sparse conversations.

Abstract

The rise of large language models (LLMs) has revolutionized the way that we interact with artificial intelligence systems through natural language. However, LLMs often misinterpret user queries because of their uncertain intention, leading to less helpful responses. In natural human interactions, clarification is sought through targeted questioning to uncover obscure information. Thus, in this paper, we introduce LaMAI (Language Model with Active Inquiry), designed to endow LLMs with this same level of interactive engagement. LaMAI leverages active learning techniques to raise the most informative questions, fostering a dynamic bidirectional dialogue. This approach not only narrows the contextual gap but also refines the output of the LLMs, aligning it more closely with user expectations. Our empirical studies, across a variety of complex datasets where LLMs have limited conversational context, demonstrate the effectiveness of LaMAI. The method improves answer accuracy from 31.9% to 50.9%, outperforming other leading question-answering frameworks. Moreover, in scenarios involving human participants, LaMAI consistently generates responses that are superior or comparable to baseline methods in more than 82% of the cases. The applicability of LaMAI is further evidenced by its successful integration with various LLMs, highlighting its potential for the future of interactive language models.

Figures (11)

• Figure 1: An illustrative example of language model active inquiry. (A) LLM directly answers the question without active inquiry. (B) LLM actively seeks clarification by inquiring about the user.
• Figure 2: Illustration of LaMAI method. (A) Overall workflow: the user query $\boldsymbol{X}$ is processed by the uncertainty estimation module. Depending on the outcome, LLM either inquires the user with clarifying questions or directly generates the response. (B) Uncertainty estimation module evaluates the LLM's level of uncertainty regarding a query $\boldsymbol{X}$. (C) In the active inquiry module, LLM actively inquires the user with clarifying questions. After receiving the user's feedback, LaMAI updates the user query to incorporate this new information and re-estimates the uncertainty. (D) Answer generation module generates the answer to a user query.
• Figure 3: Comparative and analytical results for LaMAI on the QMSum dataset: (a) GPT-4-based evaluation showing LaMAI's win rate against DG on three subsets of QMSum. (b) Human-evaluated win rate of LaMAI against DG. In this experiment, LaMAI interacts with a human participant. (c) Ablation study on the impact of the number of clarifying questions and active learning strategies on ES dataset. The values in the figure represent the win rate of LaMAI against DG method.
• Figure 4: Prompts for LaMAI.
• Figure 5: Prompt for GPT-4 evaluation.