Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM Enhancer: Merged Approach using Vector Embedding for Reducing Large Language Model Hallucinations with External Knowledge

Naheed Rayhan, Md. Ashrafuzzaman

TL;DR

This work tackles hallucinations in large language models by grounding responses in external knowledge. It introduces LLM-ENHANCER, a parallel, merged-tool architecture that ingests data from multiple web sources, encodes it with vector embeddings, and retrieves relevant chunks to inform open-source LLMs (notably Mistral 7B). The approach demonstrates substantial performance gains over sequential tool use, particularly on recent data, with higher precision and F1-scores on WikiQA and Dataset2023-24. A key contribution is showing how merging external sources via vector embeddings can reduce hallucinations without extensive fine-tuning, using openly available models and tooling to achieve reliable information retrieval and assembly.

Abstract

Large Language Models (LLMs), such as ChatGPT, have demonstrated the capability to generate human like, natural responses across a range of tasks, including task oriented dialogue and question answering. However, their application in real world, critical scenarios is often hindered by a tendency to produce inaccurate information and a limited ability to leverage external knowledge sources. This paper introduces the LLM ENHANCER system, designed to integrate multiple online sources such as Google, Wikipedia, and DuckDuckGo to enhance data accuracy. The LLMs employed within this system are open source. The data acquisition process for the LLM ENHANCER system operates in parallel, utilizing custom agent tools to manage the flow of information. Vector embeddings are used to identify the most pertinent information, which is subsequently supplied to the LLM for user interaction. The LLM ENHANCER system mitigates hallucinations in chat based LLMs while preserving response naturalness and accuracy.

LLM Enhancer: Merged Approach using Vector Embedding for Reducing Large Language Model Hallucinations with External Knowledge

TL;DR

This work tackles hallucinations in large language models by grounding responses in external knowledge. It introduces LLM-ENHANCER, a parallel, merged-tool architecture that ingests data from multiple web sources, encodes it with vector embeddings, and retrieves relevant chunks to inform open-source LLMs (notably Mistral 7B). The approach demonstrates substantial performance gains over sequential tool use, particularly on recent data, with higher precision and F1-scores on WikiQA and Dataset2023-24. A key contribution is showing how merging external sources via vector embeddings can reduce hallucinations without extensive fine-tuning, using openly available models and tooling to achieve reliable information retrieval and assembly.

Abstract

Large Language Models (LLMs), such as ChatGPT, have demonstrated the capability to generate human like, natural responses across a range of tasks, including task oriented dialogue and question answering. However, their application in real world, critical scenarios is often hindered by a tendency to produce inaccurate information and a limited ability to leverage external knowledge sources. This paper introduces the LLM ENHANCER system, designed to integrate multiple online sources such as Google, Wikipedia, and DuckDuckGo to enhance data accuracy. The LLMs employed within this system are open source. The data acquisition process for the LLM ENHANCER system operates in parallel, utilizing custom agent tools to manage the flow of information. Vector embeddings are used to identify the most pertinent information, which is subsequently supplied to the LLM for user interaction. The LLM ENHANCER system mitigates hallucinations in chat based LLMs while preserving response naturalness and accuracy.
Paper Structure (36 sections, 3 equations, 17 figures, 6 tables, 1 algorithm)

This paper contains 36 sections, 3 equations, 17 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Motivation
  4. Problem Statement
  5. Contributions
  6. Organization
  7. State-of-the-Art-Works
  8. LLM-AUGMENTER Approach
  9. Limitation of the existing works
  10. Proposed Methodology
  11. LLM-ENHANCER
  12. ZeroShot React Agent
  13. Action Executor
  14. LangChain Tools
  15. Merged Tools
  16. ...and 21 more sections

Figures (17)

  • Figure 1: Widely used Open-Source Technique.
  • Figure 2: LLM-ENHANCER.
  • Figure 3: Real Answer from the web.
  • Figure 4: Answer from the LLM without internet access
  • Figure 5: Answer from the agents where tools are Sequential
  • ...and 12 more figures