Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CAPRAG: A Large Language Model Solution for Customer Service and Automatic Reporting using Vector and Graph Retrieval-Augmented Generation

Hamza Landolsi, Kais Letaief, Nizar Taghouti, Ines Abdeljaoued-Tej

TL;DR

CAPRAG tackles information overload in digital banking by fusing Vector RAG and Graph RAG into a single Customer Analysis Pipeline RAG. It refines text data and routes queries to dual backends, with a Cypher-based graph query layer and an open-source LLM for generation. A dedicated evaluation pipeline monitors answer relevance, context relevance, and grounding, while a domain-specific knowledge graph supports geospatial and entity-linked inquiries. The approach offers improved, context-aware customer-service and reporting capabilities, operating within open-source resource constraints to empower bank customers with timely, accurate insights.

Abstract

The introduction of new features and services in the banking sector often overwhelms customers, creating an opportunity for banks to enhance user experience through financial chatbots powered by large language models (LLMs). We initiated an AI agent designed to provide customers with relevant information about banking services and insights from annual reports. We proposed a hybrid Customer Analysis Pipeline Retrieval-Augmented Generation (CAPRAG) that effectively addresses both relationship-based and contextual queries, thereby improving customer engagement in the digital banking landscape. To implement this, we developed a processing pipeline to refine text data, which we utilized in two main frameworks: Vector RAG and Graph RAG. This dual approach enables us to populate both vector and graph databases with processed data for efficient retrieval. The Cypher query component is employed to effectively query the graph database. When a user submits a query, it is first expanded by a query expansion module before being routed to construct a final query from the hybrid Knowledge Base (KB). This final query is then sent to an open-source LLM for response generation. Overall, our innovative, designed to international banks, serves bank's customers in an increasingly complex digital environment, enhancing clarity and accessibility of information.

CAPRAG: A Large Language Model Solution for Customer Service and Automatic Reporting using Vector and Graph Retrieval-Augmented Generation

TL;DR

CAPRAG tackles information overload in digital banking by fusing Vector RAG and Graph RAG into a single Customer Analysis Pipeline RAG. It refines text data and routes queries to dual backends, with a Cypher-based graph query layer and an open-source LLM for generation. A dedicated evaluation pipeline monitors answer relevance, context relevance, and grounding, while a domain-specific knowledge graph supports geospatial and entity-linked inquiries. The approach offers improved, context-aware customer-service and reporting capabilities, operating within open-source resource constraints to empower bank customers with timely, accurate insights.

Abstract

The introduction of new features and services in the banking sector often overwhelms customers, creating an opportunity for banks to enhance user experience through financial chatbots powered by large language models (LLMs). We initiated an AI agent designed to provide customers with relevant information about banking services and insights from annual reports. We proposed a hybrid Customer Analysis Pipeline Retrieval-Augmented Generation (CAPRAG) that effectively addresses both relationship-based and contextual queries, thereby improving customer engagement in the digital banking landscape. To implement this, we developed a processing pipeline to refine text data, which we utilized in two main frameworks: Vector RAG and Graph RAG. This dual approach enables us to populate both vector and graph databases with processed data for efficient retrieval. The Cypher query component is employed to effectively query the graph database. When a user submits a query, it is first expanded by a query expansion module before being routed to construct a final query from the hybrid Knowledge Base (KB). This final query is then sent to an open-source LLM for response generation. Overall, our innovative, designed to international banks, serves bank's customers in an increasingly complex digital environment, enhancing clarity and accessibility of information.
Paper Structure (14 sections, 5 figures, 3 tables)

This paper contains 14 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Data source and description of bank ${\mathcal{A}}$
  3. Evaluation pipeline: LLM as a judge
  4. Vector RAG: Methodology and Results
  5. Baseline RAG.
  6. Chunking Optimization.
  7. Enhancing Retrieval.
  8. Query Translations.
  9. Graph RAG
  10. Schema Construction
  11. Methodology
  12. Customer Analysis Pipeline RAG (CAPRAG)
  13. Discussion
  14. Conclusion

Figures (5)

  • Figure 1: Evalutation pipeline with LLM as judge
  • Figure 2: Vector RAG enhancements
  • Figure 3: Vector RAG evalutaion
  • Figure 4: Expanded graph schema using Graph Knowledge
  • Figure 5: Workflow of CAPRAG

Theorems & Definitions (3)

  • Definition 3.1: Answer Relevance
  • Definition 3.2: Context Relevance
  • Definition 3.3: Groundedness