Knowledge-Graph Based RAG System Evaluation Framework

TL;DR

This paper tackles the challenge of evaluating Retrieval Augmented Generation (RAG) systems, where traditional metrics fail to capture factuality and semantic coherence. It proposes a Knowledge Graph-based evaluation framework that extends RAGAS by constructing input/context knowledge graphs, linking them via semantic similarity, and applying multi-hop graph reasoning for factuality and content coverage. Two KG-based subscores—Multi-Hop Semantic Matching and Community-Based Semantic Overlap—are introduced, implemented via KG construction, Dijkstra path searches, and Louvain community detection, and are tested against RAGAS and human annotations across two QA datasets. Results show moderate-to-high correlations with human judgments and RAGAS, with KG methods offering enhanced sensitivity to semantic differences but facing scalability challenges, suggesting complementary use with existing evaluation frameworks and avenues for future work.

Abstract

Large language models (LLMs) has become a significant research focus and is utilized in various fields, such as text generation and dialog systems. One of the most essential applications of LLM is Retrieval Augmented Generation (RAG), which greatly enhances generated content's reliability and relevance. However, evaluating RAG systems remains a challenging task. Traditional evaluation metrics struggle to effectively capture the key features of modern LLM-generated content that often exhibits high fluency and naturalness. Inspired by the RAGAS tool, a well-known RAG evaluation framework, we extended this framework into a KG-based evaluation paradigm, enabling multi-hop reasoning and semantic community clustering to derive more comprehensive scoring metrics. By incorporating these comprehensive evaluation criteria, we gain a deeper understanding of RAG systems and a more nuanced perspective on their performance. To validate the effectiveness of our approach, we compare its performance with RAGAS scores and construct a human-annotated subset to assess the correlation between human judgments and automated metrics. In addition, we conduct targeted experiments to demonstrate that our KG-based evaluation method is more sensitive to subtle semantic differences in generated outputs. Finally, we discuss the key challenges in evaluating RAG systems and highlight potential directions for future research.

Figures (2)

• Figure 1: Comprehensive overview of all experimental results across datasets.
• Figure 2: Comparison of KG Construction and Multi-Hop Semantic Matching