Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AI Knowledge Assist: An Automated Approach for the Creation of Knowledge Bases for Conversational AI Agents

Md Tahmid Rahman Laskar, Julien Bouvier Tremblay, Xue-Yong Fu, Cheng Chen, Shashi Bhushan TN

TL;DR

AI Knowledge Assist tackles the cold-start problem in contact-center AI by automatically constructing a knowledge base from historical transcripts. It deploys a three-stage pipeline—knowledge extraction, deduplication via clustering, and recommending representative QA pairs—powered by a fine-tuned lightweight LLM (LLaMA-3.1-8B) and cost-aware components. In multi-company experiments, the end-to-end approach achieves above 90% accuracy in answering information-seeking questions and outperforms larger closed-source models, validating its practical viability for immediate RAG-powered chatbot deployment. The work also details deployment and evaluation strategies, including self-updating KB mechanisms, human-in-the-loop review, and performance analyses across model choices and ablations.

Abstract

The utilization of conversational AI systems by leveraging Retrieval Augmented Generation (RAG) techniques to solve customer problems has been on the rise with the rapid progress of Large Language Models (LLMs). However, the absence of a company-specific dedicated knowledge base is a major barrier to the integration of conversational AI systems in contact centers. To this end, we introduce AI Knowledge Assist, a system that extracts knowledge in the form of question-answer (QA) pairs from historical customer-agent conversations to automatically build a knowledge base. Fine-tuning a lightweight LLM on internal data demonstrates state-of-the-art performance, outperforming larger closed-source LLMs. More specifically, empirical evaluation on 20 companies demonstrates that the proposed AI Knowledge Assist system that leverages the LLaMA-3.1-8B model eliminates the cold-start gap in contact centers by achieving above 90% accuracy in answering information-seeking questions. This enables immediate deployment of RAG-powered chatbots.

AI Knowledge Assist: An Automated Approach for the Creation of Knowledge Bases for Conversational AI Agents

TL;DR

AI Knowledge Assist tackles the cold-start problem in contact-center AI by automatically constructing a knowledge base from historical transcripts. It deploys a three-stage pipeline—knowledge extraction, deduplication via clustering, and recommending representative QA pairs—powered by a fine-tuned lightweight LLM (LLaMA-3.1-8B) and cost-aware components. In multi-company experiments, the end-to-end approach achieves above 90% accuracy in answering information-seeking questions and outperforms larger closed-source models, validating its practical viability for immediate RAG-powered chatbot deployment. The work also details deployment and evaluation strategies, including self-updating KB mechanisms, human-in-the-loop review, and performance analyses across model choices and ablations.

Abstract

The utilization of conversational AI systems by leveraging Retrieval Augmented Generation (RAG) techniques to solve customer problems has been on the rise with the rapid progress of Large Language Models (LLMs). However, the absence of a company-specific dedicated knowledge base is a major barrier to the integration of conversational AI systems in contact centers. To this end, we introduce AI Knowledge Assist, a system that extracts knowledge in the form of question-answer (QA) pairs from historical customer-agent conversations to automatically build a knowledge base. Fine-tuning a lightweight LLM on internal data demonstrates state-of-the-art performance, outperforming larger closed-source LLMs. More specifically, empirical evaluation on 20 companies demonstrates that the proposed AI Knowledge Assist system that leverages the LLaMA-3.1-8B model eliminates the cold-start gap in contact centers by achieving above 90% accuracy in answering information-seeking questions. This enables immediate deployment of RAG-powered chatbots.

Paper Structure

This paper contains 24 sections, 3 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Our Proposed Approach
  4. Knowledge Extraction from Transcripts
  5. Clustering for Deduplication
  6. Recommending Representative QA Pairs
  7. Experimental Settings
  8. Dataset
  9. Model Selection
  10. Implementation
  11. Evaluation Settings
  12. Prompt Construction
  13. Results and Discussions
  14. Performance on Knowledge Extraction from Transcripts
  15. Clustering Results for Deduplication
  16. ...and 9 more sections

Figures (5)

  • Figure 1: An example of knowledge extracted from transcripts in the form of QA pairs.
  • Figure 2: An overview of our proposed AI Knowledge Assist. First, QA pairs are extracted from historical transcripts. Then clustering is applied to group similar QA pairs. Finally, from each cluster, representative QA pairs are constructed and then recommended for the knowledge base (a knowledge manager may review the recommended knowledge before insertion)
  • Figure 3: F1-Score per Company type for the Knowledge-Assist-8B-SFT model in terms of the Final Recommended Representative QA Pairs.
  • Figure 4: A simple demo of AI Knowledge Assist.
  • Figure 5: Performance Comparisons between Clustering Approaches.