Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IntellBot: Retrieval Augmented LLM Chatbot for Cyber Threat Knowledge Delivery

Dincy R. Arikkat, Abhinav M., Navya Binu, Parvathi M., Navya Biju, K. S. Arunima, Vinod P., Rafidha Rehiman K. A., Mauro Conti

TL;DR

IntellBot is introduced, a sophisticated cybersecurity chatbot built using advanced LLMs, Langchain, and a Retrieval-Augmented Generation model that offers intelligent responses to enhance threat intelligence, incident response, and overall security awareness.

Abstract

In the rapidly evolving landscape of cyber security, intelligent chatbots are gaining prominence. Artificial Intelligence, Machine Learning, and Natural Language Processing empower these chatbots to handle user inquiries and deliver threat intelligence. This helps cyber security knowledge readily available to both professionals and the public. Traditional rule-based chatbots often lack flexibility and struggle to adapt to user interactions. In contrast, Large Language Model-based chatbots offer contextually relevant information across multiple domains and adapt to evolving conversational contexts. In this work, we develop IntellBot, an advanced cyber security Chatbot built on top of cutting-edge technologies like Large Language Models and Langchain alongside a Retrieval-Augmented Generation model to deliver superior capabilities. This chatbot gathers information from diverse data sources to create a comprehensive knowledge base covering known vulnerabilities, recent cyber attacks, and emerging threats. It delivers tailored responses, serving as a primary hub for cyber security insights. By providing instant access to relevant information and resources, this IntellBot enhances threat intelligence, incident response, and overall security posture, saving time and empowering users with knowledge of cyber security best practices. Moreover, we analyzed the performance of our copilot using a two-stage evaluation strategy. We achieved BERT score above 0.8 by indirect approach and a cosine similarity score ranging from 0.8 to 1, which affirms the accuracy of our copilot. Additionally, we utilized RAGAS to evaluate the RAG model, and all evaluation metrics consistently produced scores above 0.77, highlighting the efficacy of our system.

IntellBot: Retrieval Augmented LLM Chatbot for Cyber Threat Knowledge Delivery

TL;DR

IntellBot is introduced, a sophisticated cybersecurity chatbot built using advanced LLMs, Langchain, and a Retrieval-Augmented Generation model that offers intelligent responses to enhance threat intelligence, incident response, and overall security awareness.

Abstract

In the rapidly evolving landscape of cyber security, intelligent chatbots are gaining prominence. Artificial Intelligence, Machine Learning, and Natural Language Processing empower these chatbots to handle user inquiries and deliver threat intelligence. This helps cyber security knowledge readily available to both professionals and the public. Traditional rule-based chatbots often lack flexibility and struggle to adapt to user interactions. In contrast, Large Language Model-based chatbots offer contextually relevant information across multiple domains and adapt to evolving conversational contexts. In this work, we develop IntellBot, an advanced cyber security Chatbot built on top of cutting-edge technologies like Large Language Models and Langchain alongside a Retrieval-Augmented Generation model to deliver superior capabilities. This chatbot gathers information from diverse data sources to create a comprehensive knowledge base covering known vulnerabilities, recent cyber attacks, and emerging threats. It delivers tailored responses, serving as a primary hub for cyber security insights. By providing instant access to relevant information and resources, this IntellBot enhances threat intelligence, incident response, and overall security posture, saving time and empowering users with knowledge of cyber security best practices. Moreover, we analyzed the performance of our copilot using a two-stage evaluation strategy. We achieved BERT score above 0.8 by indirect approach and a cosine similarity score ranging from 0.8 to 1, which affirms the accuracy of our copilot. Additionally, we utilized RAGAS to evaluate the RAG model, and all evaluation metrics consistently produced scores above 0.77, highlighting the efficacy of our system.

Paper Structure

This paper contains 21 sections, 1 equation, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Background
  4. Langchain
  5. RAG Model
  6. Methodology
  7. Data Collection
  8. Document Loading
  9. Text splitting
  10. Conversion of text into Numerical format
  11. Creation of Vector Store
  12. Similarity Search
  13. Designing Prompt
  14. Streamlit User Interface
  15. Response Evaluation
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Proposed Architecture of Security chatbot created using the Large Language Model
  • Figure 2: Comparison of average cosine similarity scores generated by Word2Vec, GloVe, and BERT embeddings related to various topics. For each topic, 100 queries were selected, and the cosine similarity between bot and human responses was computed.
  • Figure :