Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DepthCharge: A Domain-Agnostic Framework for Measuring Depth-Dependent Knowledge in Large Language Models

Alexander Sheppert

Abstract

Large Language Models appear competent when answering general questions but often fail when pushed into domain-specific details. No existing methodology provides an out-of-the-box solution for measuring how deeply LLMs can sustain accurate responses under adaptive follow-up questioning across arbitrary domains. We present DepthCharge, a domain-agnostic framework that measures knowledge depth through three innovations: adaptive probing that generates follow-up questions based on concepts the model actually mentions, on-demand fact verification from authoritative sources, and survival statistics with constant sample sizes at every depth level. The framework can be deployed on any knowledge domain with publicly verifiable facts, without requiring pre-constructed test sets or domain-specific expertise. DepthCharge results are relative to the evaluator model used for answer checking, making the framework a tool for comparative evaluation rather than absolute accuracy certification. Empirical validation across four diverse domains (Medicine, Constitutional Law, Ancient Rome, and Quantum Computing) with five frontier models demonstrates that DepthCharge reveals depth-dependent performance variation hidden by standard benchmarks. Expected Valid Depth (EVD) ranges from 3.45 to 7.55 across model-domain combinations, and model rankings vary substantially by domain, with no single model dominating all areas. Cost-performance analysis further reveals that expensive models do not always achieve deeper knowledge, suggesting that domain-specific evaluation is more informative than aggregate benchmarks for model selection in professional applications.

DepthCharge: A Domain-Agnostic Framework for Measuring Depth-Dependent Knowledge in Large Language Models

Abstract

Large Language Models appear competent when answering general questions but often fail when pushed into domain-specific details. No existing methodology provides an out-of-the-box solution for measuring how deeply LLMs can sustain accurate responses under adaptive follow-up questioning across arbitrary domains. We present DepthCharge, a domain-agnostic framework that measures knowledge depth through three innovations: adaptive probing that generates follow-up questions based on concepts the model actually mentions, on-demand fact verification from authoritative sources, and survival statistics with constant sample sizes at every depth level. The framework can be deployed on any knowledge domain with publicly verifiable facts, without requiring pre-constructed test sets or domain-specific expertise. DepthCharge results are relative to the evaluator model used for answer checking, making the framework a tool for comparative evaluation rather than absolute accuracy certification. Empirical validation across four diverse domains (Medicine, Constitutional Law, Ancient Rome, and Quantum Computing) with five frontier models demonstrates that DepthCharge reveals depth-dependent performance variation hidden by standard benchmarks. Expected Valid Depth (EVD) ranges from 3.45 to 7.55 across model-domain combinations, and model rankings vary substantially by domain, with no single model dominating all areas. Cost-performance analysis further reveals that expensive models do not always achieve deeper knowledge, suggesting that domain-specific evaluation is more informative than aggregate benchmarks for model selection in professional applications.
Paper Structure (50 sections, 3 equations, 3 figures, 10 tables)

This paper contains 50 sections, 3 equations, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Motivation
  3. The Surface Competence Illusion
  4. Framework Overview
  5. Contributions
  6. Related Work
  7. LLM Benchmarks and Evaluation
  8. Cognitive and Taxonomic Evaluation
  9. Multi-Turn and Adaptive Evaluation
  10. Medical and Domain-Specific Benchmarks
  11. Hallucination and Factuality
  12. Question Generation
  13. Positioning of DepthCharge
  14. Methodology
  15. Domain Requirements
  16. ...and 35 more sections

Figures (3)

  • Figure 1: Tree diagram showing adaptive drilling with cumulative survival calculation. Green nodes indicate correct answers (paths continue), red nodes indicate incorrect answers (paths terminate). Cumulative survival at each depth equals the product of all accuracies up to that point, reflecting that errors permanently reduce survival probability.
  • Figure 2: Example drilling path showing question-answer pairs at increasing depths. D1-D3 (COMMON tier) ask increasingly specific questions about foundational concepts. D4-D6 (TEXTBOOK tier) probe detailed knowledge requiring deeper familiarity with the subject. Each question builds on concepts from the previous answer.
  • Figure 3: Comprehensive comparison across all model-domain combinations. Top left: EVD heatmap showing performance variation. Top right: Average EVD rankings. Bottom left: Survival scatter comparing top performers. Bottom right: Value analysis (EVD vs cost).