Enhancing Discoverability in Enterprise Conversational Systems with Proactive Question Suggestions

TL;DR

The paper tackles the problem of aiding new users in enterprise conversational AI to ask effective questions and discover platform capabilities. It introduces a two-stage framework that combines population-level user intent analysis with chat-session level question generation, powered by LLMs and retrieval-augmented data. Evaluation on real-world Adobe Experience Platform AI Assistant data, including a human study, shows improvements in usefulness and discoverability of suggested questions. The work advances enterprise assistants by enabling proactive, context-aware guidance that helps users explore features and adopt the system more effectively.

Abstract

Enterprise conversational AI systems are becoming increasingly popular to assist users in completing daily tasks such as those in marketing and customer management. However, new users often struggle to ask effective questions, especially in emerging systems with unfamiliar or evolving capabilities. This paper proposes a framework to enhance question suggestions in conversational enterprise AI systems by generating proactive, context-aware questions that try to address immediate user needs while improving feature discoverability. Our approach combines periodic user intent analysis at the population level with chat session-based question generation. We evaluate the framework using real-world data from the AI Assistant for Adobe Experience Platform (AEP), demonstrating the improved usefulness and system discoverability of the AI Assistant.

Figures (4)

• Figure 1: Next Question Suggestion Framework in Enterprise Conversational AI Systems. The framework consists of two components: population-level user intent analysis (right), which generates question categories used for next-question suggestion at the chat-session level (middle). The response generation process (left) is included for completeness, as it precedes the question suggestion step and shares several inputs and outputs, such as the user query, retrieved documents, and AI response.
• Figure 2: Prompt template used for generating contextual and categorized question suggestions. Certain details have been omitted due to space constraints or business confidentiality.
• Figure 3: Example of question suggestions in AEP’s AI Assistant. When the user asks a question, the AI Assistant first provides a response to the question, followed by two suggested questions of different categories, with one expanding the related concepts mentioned in the question and response, and one suggesting potential future steps that can be taken to better utilize the platform functionalities.
• Figure 4: Human Evaluation Interface. Annotators used this interface to compare the quality of question suggestions based on the defined criteria.