Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated Retrieval Augmented Generation for Multi-Product Question Answering

Parshin Shojaee, Sai Sree Harsha, Dan Luo, Akash Maharaj, Tong Yu, Yunyao Li

TL;DR

The paper tackles multi-product QA in enterprise contexts by proposing MKP-QA, a probabilistic federated search framework that jointly models query-domain relevance and cross-domain document relevance. It introduces a Domain Router, a Stochastic Gating mechanism, a bi-encoder Retriever, and federated aggregation to produce a unified, domain-aware retrieval set for RAG-based answer generation, avoiding monolithic centralization. New Adobe ExL-based uni- and cross-domain benchmarks for AEP, Target, and CJA are provided to evaluate retrieval accuracy and response quality, with MKP-QA showing significant gains, especially in cross-domain scenarios, without requiring domain-specific LLM fine-tuning. The work demonstrates practical improvements and outlines deployment considerations, including knowledge precision, latency, user studies, and continuous monitoring to ensure robust real-world performance in enterprise product QA.

Abstract

Recent advancements in Large Language Models and Retrieval-Augmented Generation have boosted interest in domain-specific question-answering for enterprise products. However, AI Assistants often face challenges in multi-product QA settings, requiring accurate responses across diverse domains. Existing multi-domain RAG-QA approaches either query all domains indiscriminately, increasing computational costs and LLM hallucinations, or rely on rigid resource selection, which can limit search results. We introduce MKP-QA, a novel multi-product knowledge-augmented QA framework with probabilistic federated search across domains and relevant knowledge. This method enhances multi-domain search quality by aggregating query-domain and query-passage probabilistic relevance. To address the lack of suitable benchmarks for multi-product QAs, we also present new datasets focused on three Adobe products: Adobe Experience Platform, Target, and Customer Journey Analytics. Our experiments show that MKP-QA significantly boosts multi-product RAG-QA performance in terms of both retrieval accuracy and response quality.

Federated Retrieval Augmented Generation for Multi-Product Question Answering

TL;DR

The paper tackles multi-product QA in enterprise contexts by proposing MKP-QA, a probabilistic federated search framework that jointly models query-domain relevance and cross-domain document relevance. It introduces a Domain Router, a Stochastic Gating mechanism, a bi-encoder Retriever, and federated aggregation to produce a unified, domain-aware retrieval set for RAG-based answer generation, avoiding monolithic centralization. New Adobe ExL-based uni- and cross-domain benchmarks for AEP, Target, and CJA are provided to evaluate retrieval accuracy and response quality, with MKP-QA showing significant gains, especially in cross-domain scenarios, without requiring domain-specific LLM fine-tuning. The work demonstrates practical improvements and outlines deployment considerations, including knowledge precision, latency, user studies, and continuous monitoring to ensure robust real-world performance in enterprise product QA.

Abstract

Recent advancements in Large Language Models and Retrieval-Augmented Generation have boosted interest in domain-specific question-answering for enterprise products. However, AI Assistants often face challenges in multi-product QA settings, requiring accurate responses across diverse domains. Existing multi-domain RAG-QA approaches either query all domains indiscriminately, increasing computational costs and LLM hallucinations, or rely on rigid resource selection, which can limit search results. We introduce MKP-QA, a novel multi-product knowledge-augmented QA framework with probabilistic federated search across domains and relevant knowledge. This method enhances multi-domain search quality by aggregating query-domain and query-passage probabilistic relevance. To address the lack of suitable benchmarks for multi-product QAs, we also present new datasets focused on three Adobe products: Adobe Experience Platform, Target, and Customer Journey Analytics. Our experiments show that MKP-QA significantly boosts multi-product RAG-QA performance in terms of both retrieval accuracy and response quality.
Paper Structure (32 sections, 1 equation, 9 figures, 1 table)

This paper contains 32 sections, 1 equation, 9 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Domain-specific Question-Answering
  4. Retrieval Augmented Generation
  5. Multi-domain Document Retrieval
  6. Methodology
  7. Query-Domain Router
  8. Stochastic Gating
  9. Query-Document Retriever
  10. Federated Search
  11. Dataset Creation and Statistics
  12. Data Sources
  13. Data Pre-processing
  14. Data Creation
  15. Data Analysis and Statistics
  16. ...and 17 more sections

Figures (9)

  • Figure 1: Overview of the MKP-QA framework.ⓐ The main RAG-QA pipeline: retrieval of multi-domain documents, prompt augmentation, and response generation. ⓑ Detailed view of the multi-domain knowledge augmentation: A domain router estimates query-domain relevance while a retriever finds relevant documents across product domains. A stochastic gating mechanism determines active domains, for which query-domain and query-document relevance scores are aggregated into a unified ranking of multi-domain documents. These top-ranked documents then augment the prompt, enabling effective cross-domain product QA.
  • Figure 2: Performance comparison of retrieval accuracy (Top-$1$) across methods on (a) uni-domain, and (b) cross-domain datasets.
  • Figure 3: Performance comparison of response quality across methods on different datasets for LLM-based (a) Relevancy, and (b) Faithfulness metrics.
  • Figure 4: Examples of user query and relevant product documentation in the dataset.
  • Figure 5: LLM Prompt for Query Generation: Simulating user behavior for document-based question synthesis
  • ...and 4 more figures