Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Retrieval Augmented Generation-Based Incident Resolution Recommendation System for IT Support

Paulina Toro Isaza, Michael Nidd, Noah Zheutlin, Jae-wook Ahn, Chidansh Amitkumar Bhatt, Yu Deng, Ruchi Mahindru, Martin Franz, Hans Florian, Salim Roukos

TL;DR

This work presents a system developed for a client in the IT Support domain for support case solution recommendation that combines retrieval augmented generation (RAG) for answer generation with an encoder-only model for classification and a generative large language model for query generation.

Abstract

Clients wishing to implement generative AI in the domain of IT Support and AIOps face two critical issues: domain coverage and model size constraints due to model choice limitations. Clients might choose to not use larger proprietary models such as GPT-4 due to cost and privacy concerns and so are limited to smaller models with potentially less domain coverage that do not generalize to the client's domain. Retrieval augmented generation is a common solution that addresses both of these issues: a retrieval system first retrieves the necessary domain knowledge which a smaller generative model leverages as context for generation. We present a system developed for a client in the IT Support domain for support case solution recommendation that combines retrieval augmented generation (RAG) for answer generation with an encoder-only model for classification and a generative large language model for query generation. We cover architecture details, data collection and annotation, development journey and preliminary validations, expected final deployment process and evaluation plans, and finally lessons learned.

Retrieval Augmented Generation-Based Incident Resolution Recommendation System for IT Support

TL;DR

This work presents a system developed for a client in the IT Support domain for support case solution recommendation that combines retrieval augmented generation (RAG) for answer generation with an encoder-only model for classification and a generative large language model for query generation.

Abstract

Clients wishing to implement generative AI in the domain of IT Support and AIOps face two critical issues: domain coverage and model size constraints due to model choice limitations. Clients might choose to not use larger proprietary models such as GPT-4 due to cost and privacy concerns and so are limited to smaller models with potentially less domain coverage that do not generalize to the client's domain. Retrieval augmented generation is a common solution that addresses both of these issues: a retrieval system first retrieves the necessary domain knowledge which a smaller generative model leverages as context for generation. We present a system developed for a client in the IT Support domain for support case solution recommendation that combines retrieval augmented generation (RAG) for answer generation with an encoder-only model for classification and a generative large language model for query generation. We cover architecture details, data collection and annotation, development journey and preliminary validations, expected final deployment process and evaluation plans, and finally lessons learned.
Paper Structure (17 sections, 3 figures, 6 tables)

This paper contains 17 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Architecture
  3. Data
  4. Development and Validation
  5. Case Turn Classifier
  6. Query Generator
  7. Retriever
  8. Answer Generator
  9. Deployment
  10. Lessons Learned
  11. Use Case Formation
  12. Inter-Annotator Agreement
  13. RAG Bottlenecks
  14. Related Work
  15. LLM-Based AIOps
  16. ...and 2 more sections

Figures (3)

  • Figure 1: System architecture
  • Figure 2: Retriever recall for top $x$ (log scale), comparing with (yellow) and without (orange) re-ranking vs. direct Google search via SerpApi (grey) for 1729 customer issues over six products
  • Figure 3: Working mockup of online deployment UI of single system result with feedback items.