Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Models

Declan Jackson, William Keating, George Cameron, Micah Hill-Smith

TL;DR

This paper introduces AA-Omniscience, a cross-domain benchmark designed to measure both factual recall and knowledge calibration across 42 topics in six economically important domains, using 6,000 questions derived from authoritative sources. It defines the Omniscience Index, a bounded metric from $-100$ to $100$ that strongly rewards abstention and penalizes incorrect guesses, alongside accuracy, hallucination rate, and cost to run, enabling a holistic view of knowledge reliability and deployment cost. An automated question-generation pipeline generates and revises questions to ensure difficulty, unambiguity, and current relevance, producing granular results across domains and topics. Findings show persistent factuality and calibration weaknesses among frontier models, with domain-specific patterns and cost considerations underscoring the need to select models based on use-case demands rather than overall performance alone.

Abstract

Existing language model evaluations primarily measure general capabilities, yet reliable use of these models across a range of domains demands factual accuracy and recognition of knowledge gaps. We introduce AA-Omniscience, a benchmark designed to measure both factual recall and knowledge calibration across 6,000 questions. Questions are derived from authoritative academic and industry sources, and cover 42 economically relevant topics within six different domains. The evaluation measures a model's Omniscience Index, a bounded metric (-100 to 100) measuring factual recall that jointly penalizes hallucinations and rewards abstention when uncertain, with 0 equating to a model that answers questions correctly as much as it does incorrectly. Among evaluated models, Claude 4.1 Opus attains the highest score (4.8), making it one of only three models to score above zero. These results reveal persistent factuality and calibration weaknesses across frontier models. Performance also varies by domain, with the models from three different research labs leading across the six domains. This performance variability suggests models should be chosen according to the demands of the use case rather than general performance for tasks where knowledge is important.

AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Models

TL;DR

This paper introduces AA-Omniscience, a cross-domain benchmark designed to measure both factual recall and knowledge calibration across 42 topics in six economically important domains, using 6,000 questions derived from authoritative sources. It defines the Omniscience Index, a bounded metric from to that strongly rewards abstention and penalizes incorrect guesses, alongside accuracy, hallucination rate, and cost to run, enabling a holistic view of knowledge reliability and deployment cost. An automated question-generation pipeline generates and revises questions to ensure difficulty, unambiguity, and current relevance, producing granular results across domains and topics. Findings show persistent factuality and calibration weaknesses among frontier models, with domain-specific patterns and cost considerations underscoring the need to select models based on use-case demands rather than overall performance alone.

Abstract

Existing language model evaluations primarily measure general capabilities, yet reliable use of these models across a range of domains demands factual accuracy and recognition of knowledge gaps. We introduce AA-Omniscience, a benchmark designed to measure both factual recall and knowledge calibration across 6,000 questions. Questions are derived from authoritative academic and industry sources, and cover 42 economically relevant topics within six different domains. The evaluation measures a model's Omniscience Index, a bounded metric (-100 to 100) measuring factual recall that jointly penalizes hallucinations and rewards abstention when uncertain, with 0 equating to a model that answers questions correctly as much as it does incorrectly. Among evaluated models, Claude 4.1 Opus attains the highest score (4.8), making it one of only three models to score above zero. These results reveal persistent factuality and calibration weaknesses across frontier models. Performance also varies by domain, with the models from three different research labs leading across the six domains. This performance variability suggests models should be chosen according to the demands of the use case rather than general performance for tasks where knowledge is important.

Paper Structure

This paper contains 23 sections, 3 equations, 12 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Topic selection and data discovery
  4. Question generation
  5. Question answering and grading methodology
  6. Metrics
  7. Omniscience Index
  8. Accuracy
  9. Hallucination rate
  10. Cost to run
  11. Results
  12. Overall results
  13. Omniscience Index and Model Intelligence
  14. Cross-domain performance
  15. Omniscience relationship with model size
  16. ...and 8 more sections

Figures (12)

  • Figure 1: Omniscience Index results
  • Figure 2: Real-world chat application example demonstrating poorer outcomes from models with limited embedded knowledge relative to models with strong embedded knowledge capabilities
  • Figure 3: Distribution of the 6,000 question AA-Omniscience dataset across the 6 domains and 42 categories
  • Figure 4: Distribution of first answerable date by domain
  • Figure 5: Omniscience Accuracy vs. Omniscience Index
  • ...and 7 more figures