Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universe

JV Roig

TL;DR

RIKER tackles core evaluation challenges for enterprise knowledge retrieval by inverting the traditional ground-truth workflow: it generates documents from a structured ground truth to enable deterministic scoring and regenerable corpora. The approach combines synthetic data generation, a coherent universe of entities, a multi-level question taxonomy, and rigorous fidelity metrics to separate grounding, fabrication, and aggregation capabilities, validated through cross-corpus stability. Empirical results across 33 models show strong performance at moderate contexts but clear degradation with longer contexts, with aggregation proving substantially harder than single-document extraction and grounding not predicting fabrication propensity. The methodology offers a domain-agnostic, contamination-resistant framework that can be applied to retrieval, RAG, and KG systems, enabling scalable benchmarking and reproducible comparisons in real-world enterprise settings.

Abstract

Evaluating knowledge systems (LLMs, RAG, knowledge graphs, etc) faces fundamental challenges: static benchmarks are vulnerable to contamination, LLM-based judges exhibit systematic biases, and ground truth extraction requires expensive human annotation. We present RIKER (Retrieval Intelligence and Knowledge Extraction Rating), both a benchmark and a replicable methodology based on paradigm inversion - generating documents from known ground truth rather than extracting ground truth from documents. This approach enables deterministic scoring and scalable evaluation without human annotation or reference models, and contamination resistance through regenerable corpora. Our evaluation of 33 models using over 21 billion tokens reveals that context length claims frequently exceed usable capacity, with significant degradation beyond 32K tokens; cross-document aggregation proves substantially harder than single-document extraction; and grounding ability and hallucination resistance are distinct capabilities - models excelling at finding facts that exist may still fabricate facts that do not. Beyond the specific benchmark, we contribute a domain-agnostic methodology for constructing scalable and contamination-resistant evaluations wherever synthetic documents can be generated from structured ground truth.

Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universe

TL;DR

RIKER tackles core evaluation challenges for enterprise knowledge retrieval by inverting the traditional ground-truth workflow: it generates documents from a structured ground truth to enable deterministic scoring and regenerable corpora. The approach combines synthetic data generation, a coherent universe of entities, a multi-level question taxonomy, and rigorous fidelity metrics to separate grounding, fabrication, and aggregation capabilities, validated through cross-corpus stability. Empirical results across 33 models show strong performance at moderate contexts but clear degradation with longer contexts, with aggregation proving substantially harder than single-document extraction and grounding not predicting fabrication propensity. The methodology offers a domain-agnostic, contamination-resistant framework that can be applied to retrieval, RAG, and KG systems, enabling scalable benchmarking and reproducible comparisons in real-world enterprise settings.

Abstract

Evaluating knowledge systems (LLMs, RAG, knowledge graphs, etc) faces fundamental challenges: static benchmarks are vulnerable to contamination, LLM-based judges exhibit systematic biases, and ground truth extraction requires expensive human annotation. We present RIKER (Retrieval Intelligence and Knowledge Extraction Rating), both a benchmark and a replicable methodology based on paradigm inversion - generating documents from known ground truth rather than extracting ground truth from documents. This approach enables deterministic scoring and scalable evaluation without human annotation or reference models, and contamination resistance through regenerable corpora. Our evaluation of 33 models using over 21 billion tokens reveals that context length claims frequently exceed usable capacity, with significant degradation beyond 32K tokens; cross-document aggregation proves substantially harder than single-document extraction; and grounding ability and hallucination resistance are distinct capabilities - models excelling at finding facts that exist may still fabricate facts that do not. Beyond the specific benchmark, we contribute a domain-agnostic methodology for constructing scalable and contamination-resistant evaluations wherever synthetic documents can be generated from structured ground truth.
Paper Structure (57 sections, 1 figure, 16 tables)

This paper contains 57 sections, 1 figure, 16 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Long-Context LLM Evaluation
  4. Retrieval and Embedding Benchmarks
  5. Multi-Hop Question Answering
  6. RAG Evaluation
  7. GraphRAG and Knowledge Graph Evaluation
  8. Hallucination and Factuality Benchmarks
  9. Benchmark Contamination
  10. Synthetic Data Generation and Simulation Validity
  11. The Bias Factor Problem
  12. The Reality Gap
  13. Parameterized Evaluation
  14. The PICARD Framework
  15. Summary: Research Gaps
  16. ...and 42 more sections

Figures (1)

  • Figure 1: RIKER methodology overview. Traditional approaches (left) extract ground truth from existing documents via expensive human annotation, producing static benchmarks vulnerable to contamination and requiring biased LLM judges. RIKER (right) inverts this: structured ground truth is defined first, then documents and questions are generated from it, enabling deterministic scoring and regenerable corpora.