Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RECKON: Large-scale Reference-based Efficient Knowledge Evaluation for Large Language Model

Lin Zhang, Zhouhong Gu, Xiaoran Shi, Hongwei Feng, Yanghua Xiao

TL;DR

RecKon introduces a reference-based knowledge evaluation framework that decomposes large, unstructured reference data into knowledge units, clusters them, and generates targeted questions to assess coverage across diverse domains. By relying on external references rather than model-derived answers, RecKon achieves high alignment with human judgments while reducing evaluation costs by over 56% and maintaining accuracy above 97% in world knowledge, code, legal, and biomedical tasks. The framework demonstrates strong cross-domain applicability, improved stability with references, and insightful analysis on clustering and question-type performance, though it depends on the quality of external references and careful prompt design. Overall, RecKon offers a scalable, objective, and adaptable pathway for robust knowledge evaluation of large language systems, with practical implications for benchmarking, bias mitigation, and cost-efficient assessment.

Abstract

As large language models (LLMs) advance, efficient knowledge evaluation becomes crucial to verifying their capabilities. Traditional methods, relying on benchmarks, face limitations such as high resource costs and information loss. We propose the Large-scale Reference-based Efficient Knowledge Evaluation for Large Language Model (RECKON), which directly uses reference data to evaluate models. RECKON organizes unstructured data into manageable units and generates targeted questions for each cluster, improving evaluation accuracy and efficiency. Experimental results show that RECKON reduces resource consumption by 56.5% compared to traditional methods while achieving over 97% accuracy across various domains, including world knowledge, code, legal, and biomedical datasets. Code is available at https://github.com/MikeGu721/reckon

RECKON: Large-scale Reference-based Efficient Knowledge Evaluation for Large Language Model

TL;DR

RecKon introduces a reference-based knowledge evaluation framework that decomposes large, unstructured reference data into knowledge units, clusters them, and generates targeted questions to assess coverage across diverse domains. By relying on external references rather than model-derived answers, RecKon achieves high alignment with human judgments while reducing evaluation costs by over 56% and maintaining accuracy above 97% in world knowledge, code, legal, and biomedical tasks. The framework demonstrates strong cross-domain applicability, improved stability with references, and insightful analysis on clustering and question-type performance, though it depends on the quality of external references and careful prompt design. Overall, RecKon offers a scalable, objective, and adaptable pathway for robust knowledge evaluation of large language systems, with practical implications for benchmarking, bias mitigation, and cost-efficient assessment.

Abstract

As large language models (LLMs) advance, efficient knowledge evaluation becomes crucial to verifying their capabilities. Traditional methods, relying on benchmarks, face limitations such as high resource costs and information loss. We propose the Large-scale Reference-based Efficient Knowledge Evaluation for Large Language Model (RECKON), which directly uses reference data to evaluate models. RECKON organizes unstructured data into manageable units and generates targeted questions for each cluster, improving evaluation accuracy and efficiency. Experimental results show that RECKON reduces resource consumption by 56.5% compared to traditional methods while achieving over 97% accuracy across various domains, including world knowledge, code, legal, and biomedical datasets. Code is available at https://github.com/MikeGu721/reckon

Paper Structure

This paper contains 32 sections, 9 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. RecKon Framework
  4. Experiment Setup
  5. Dataset and models
  6. Baseline
  7. Performance
  8. LLM-based methods exhibit significantly higher evaluation accuracy compared to both n-gram-based and embedding-based approaches.
  9. External references enhance evaluation accuracy and stability by countering internal model biases.
  10. RecKon reduces the cost of evaluation and maintains competitive correlation with Jw/R.
  11. Analysis
  12. Evaluation Results
  13. Evaluation Performance with Reference Data
  14. Evaluation Performance Knowledge Clustering and Concept Characteristics
  15. Evaluation Performance of Different Types of Questions
  16. ...and 17 more sections

Figures (9)

  • Figure 1: The comparison of different evaluation method.
  • Figure 2: RecKon consists of five main components that form a cycle of assessment.
  • Figure 3: The counts of 6 types of questions of RecKon used for different models.
  • Figure 4: The embedding map of the remaining knowledge units to be evaluated on the Wiki dataset in the process of evaluating GPT3.5-turbo at round k (k is different of the two sub-figures), where the representation of knowledge units of the same color is clustered in the same class, and their class names are displayed.
  • Figure 5: The frequency of knowledge units in multiple iterations of RecKon. The X-axis is the frequency of knowledge units, and the vertical axis is the number of knowledge units of a certain frequency.
  • ...and 4 more figures