Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Query Suggestion for Retrieval-Augmented Generation via Dynamic In-Context Learning

Fabian Spaeh, Tianyi Chen, Chen-Hao Chiang, Bin Shen

TL;DR

This work addresses the challenge of guiding user interactions with agentic RAG when initial queries are unanswerable by proposing query suggestions that preserve intent and are executable by the RAG workflow. It introduces dynamic few-shot learning with a templating-based workflow mapping, robust dynamic retrieval, and self-learning to label data without manual annotations. Empirical results on three real-world RAG agents show that the proposed method improves both the relevance (semantic similarity) and answerability of suggested queries compared to static few-shot and retrieval-only baselines. The approach enables safer, more effective interaction with complex, multi-step agentic systems and offers scalable building blocks for adaptive prompting and workflow inspection in operational AI.

Abstract

Retrieval-augmented generation with tool-calling agents (agentic RAG) has become increasingly powerful in understanding, processing, and responding to user queries. However, the scope of the grounding knowledge is limited and asking questions that exceed this scope may lead to issues like hallucination. While guardrail frameworks aim to block out-of-scope questions (Rodriguez et al., 2024), no research has investigated the question of suggesting answerable queries in order to complete the user interaction. In this paper, we initiate the study of query suggestion for agentic RAG. We consider the setting where user questions are not answerable, and the suggested queries should be similar to aid the user interaction. Such scenarios are frequent for tool-calling LLMs as communicating the restrictions of the tools or the underlying datasets to the user is difficult, and adding query suggestions enhances the interaction with the RAG agent. As opposed to traditional settings for query recommendations such as in search engines, ensuring that the suggested queries are answerable is a major challenge due to the RAG's multi-step workflow that demands a nuanced understanding of the RAG as a whole, which the executing LLM lacks. As such, we introduce robust dynamic few-shot learning which retrieves examples from relevant workflows. We show that our system can be self-learned, for instance on prior user queries, and is therefore easily applicable in practice. We evaluate our approach on three benchmark datasets based on two unlabeled question datasets collected from real-world user queries. Experiments on real-world datasets confirm that our method produces more relevant and answerable suggestions, outperforming few-shot and retrieval-only baselines, and thus enable safer, more effective user interaction with agentic RAG.

Query Suggestion for Retrieval-Augmented Generation via Dynamic In-Context Learning

TL;DR

This work addresses the challenge of guiding user interactions with agentic RAG when initial queries are unanswerable by proposing query suggestions that preserve intent and are executable by the RAG workflow. It introduces dynamic few-shot learning with a templating-based workflow mapping, robust dynamic retrieval, and self-learning to label data without manual annotations. Empirical results on three real-world RAG agents show that the proposed method improves both the relevance (semantic similarity) and answerability of suggested queries compared to static few-shot and retrieval-only baselines. The approach enables safer, more effective interaction with complex, multi-step agentic systems and offers scalable building blocks for adaptive prompting and workflow inspection in operational AI.

Abstract

Retrieval-augmented generation with tool-calling agents (agentic RAG) has become increasingly powerful in understanding, processing, and responding to user queries. However, the scope of the grounding knowledge is limited and asking questions that exceed this scope may lead to issues like hallucination. While guardrail frameworks aim to block out-of-scope questions (Rodriguez et al., 2024), no research has investigated the question of suggesting answerable queries in order to complete the user interaction. In this paper, we initiate the study of query suggestion for agentic RAG. We consider the setting where user questions are not answerable, and the suggested queries should be similar to aid the user interaction. Such scenarios are frequent for tool-calling LLMs as communicating the restrictions of the tools or the underlying datasets to the user is difficult, and adding query suggestions enhances the interaction with the RAG agent. As opposed to traditional settings for query recommendations such as in search engines, ensuring that the suggested queries are answerable is a major challenge due to the RAG's multi-step workflow that demands a nuanced understanding of the RAG as a whole, which the executing LLM lacks. As such, we introduce robust dynamic few-shot learning which retrieves examples from relevant workflows. We show that our system can be self-learned, for instance on prior user queries, and is therefore easily applicable in practice. We evaluate our approach on three benchmark datasets based on two unlabeled question datasets collected from real-world user queries. Experiments on real-world datasets confirm that our method produces more relevant and answerable suggestions, outperforming few-shot and retrieval-only baselines, and thus enable safer, more effective user interaction with agentic RAG.
Paper Structure (28 sections, 1 equation, 10 figures, 1 algorithm)

This paper contains 28 sections, 1 equation, 10 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Our Contributions and Techniques
  3. Related Work
  4. Query Suggestion
  5. LLM Guardrails
  6. Query Disambiguation
  7. Organization
  8. Query Suggestion for RAG
  9. Our Approach
  10. Dynamic Few-Shot Learning
  11. Question Templating
  12. Dynamic Few-Shot Retrieval
  13. Generation
  14. Self-Learning
  15. Response Evaluation
  16. ...and 13 more sections

Figures (10)

  • Figure 1: Two queries that are not answered. Black boxes denote a tool execution. (Left) the RAG agent does not have access to information about orders, i.e. it is out of scope and there is no workflow that will successfully answer the query. (Right) The query fails since the final tool call returns an empty response due to a typo in the year, indicating that there is no underlying knowledge about the year 2042.
  • Figure 2: Dynamic Few-Shot Learning. Positive examples are labeled queries that are answerable; negative examples are queries for which there is no workflow.
  • Figure 3: Simultaneous learning and query suggestion. The LLM evaluates the answerability of the query based on the chain of tool-calls and response produced by the RAG agent. The evaluation is stored in the similarity database using the embedding vector of the templated query. At the same time, we use the template query for dynamic retrieval of few-shot examples.
  • Figure 4: Prompt for the generation of an answerable template query. For brevity, we do not show the full prompt and explanations for answerable example queries. The explanation for the second negative example gives a strong hint that Python code will not be executed, so ad-hoc calculations are not allowed for this instance. As a result, we obtain the suggestion "What is the total number of invoices paid late in [timespan]?" which can indeed be answered.
  • Figure 5: Dataset statistics. We show the number of queries along with their classification, as described in Section \ref{['sec:ans-eval']}.
  • ...and 5 more figures