Redefining Proactivity for Information Seeking Dialogue

TL;DR

A new definition of proactivity is presented that focuses on enhancing the `proactiveness' of each generated response via the introduction of new information related to the initial query via the introduction of new information related to the initial query.

Abstract

Information-Seeking Dialogue (ISD) agents aim to provide accurate responses to user queries. While proficient in directly addressing user queries, these agents, as well as LLMs in general, predominantly exhibit reactive behavior, lacking the ability to generate proactive responses that actively engage users in sustained conversations. However, existing definitions of proactive dialogue in this context do not focus on how each response actively engages the user and sustains the conversation. Hence, we present a new definition of proactivity that focuses on enhancing the `proactiveness' of each generated response via the introduction of new information related to the initial query. To this end, we construct a proactive dialogue dataset comprising 2,000 single-turn conversations, and introduce several automatic metrics to evaluate response `proactiveness' which achieved high correlation with human annotation. Additionally, we introduce two innovative Chain-of-Thought (CoT) prompts, the 3-step CoT and the 3-in-1 CoT prompts, which consistently outperform standard prompts by up to 90% in the zero-shot setting.

Figures (8)

• Figure 1: Illustration of a single user query and the corresponding reactive and proactive responses. Each proactive response corresponds to a specific proactive element type. The follow-up question is marked in orange text, additional information is denoted by green text, and the answer component is indicated in blue text.
• Figure 2: Examples of FQs and AI. Proactive elements that are accepted or unaccepted are symbolized by a green checkmark or a red "X" respectively. The criteria for deeming each proactive element as unacceptable is specified adjacent to the corresponding red "X".
• Figure 3: (a) Illustration of low and high semantic similarities in low quality AI and FQ respectively. (b) Samples of LLM-generated user responses for AI. (c) Samples of LLM-generated user responses for FQ.
• Figure 4: AMT instructions for the FQ.
• Figure 5: AMT instructions for the AI.