Dynamic Contexts for Generating Suggestion Questions in RAG Based Conversational Systems

TL;DR

The paper addresses the difficulty users face when querying RAG-based conversational agents by introducing Dynamic Contexts, a framework that generates three suggestion questions guided by the user's initial query. It combines dynamic few-shot examples with dynamically retrieved contexts, selecting components via OpenAI embeddings and cosine similarity to produce an effective prompt for question generation. The approach demonstrates improvements over zero-shot and static few-shot prompting on a practical dataset, with comprehensive evaluations including manual, comparative, and preference analyses. This work advances practical, low-resource augmentation of RAG systems by reducing ambiguity and enhancing conversational flow through targeted clarification questions.

Abstract

When interacting with Retrieval-Augmented Generation (RAG)-based conversational agents, the users must carefully craft their queries to be understood correctly. Yet, understanding the system's capabilities can be challenging for the users, leading to ambiguous questions that necessitate further clarification. This work aims to bridge the gap by developing a suggestion question generator. To generate suggestion questions, our approach involves utilizing dynamic context, which includes both dynamic few-shot examples and dynamically retrieved contexts. Through experiments, we show that the dynamic contexts approach can generate better suggestion questions as compared to other prompting approaches.

Figures (1)

• Figure 1: Dynamic Context: Suggestion Questions are generated in RAG based Chatbots using the initial user query along with dynamically few-shot examples and dynamic retrieved contexts