Understanding Understanding: A Pragmatic Framework Motivated by Large Language Models
Kevin Leyton-Brown, Yoav Shoham
TL;DR
The paper defines a formal, pragmatic notion of understanding based on a domain-specific scope of questions and a dual criterion: achieving a high average answer quality $\mathbb{E}_{q\sim\Delta_Q} S(q,\text{ans}(q)) \geq PG$ while keeping the probability of ridiculous answers below $RID$. It develops a statistically principled testing regimen using independent sampling and Chernoff-derived bounds to certify understanding with high confidence, and analyzes the associated sample complexity. Recognizing the cost of stringent ridiculousness constraints, it introduces explanations as auxiliary mechanisms: when agents provide explanations via applicable procedures, the effective question space expands combinatorially, enabling tighter confidence with fewer samples. The framework culminates in formal guarantees and practical guidance for evaluating and designing AI agents that can genuinely understand, while noting that modern LLMs fall short in nontrivial domains under these criteria. Overall, the work offers a rigorous, testable recipe for understanding and a path toward more robust, explainable AI systems.
Abstract
Motivated by the rapid ascent of Large Language Models (LLMs) and debates about the extent to which they possess human-level qualities, we propose a framework for testing whether any agent (be it a machine or a human) understands a subject matter. In Turing-test fashion, the framework is based solely on the agent's performance, and specifically on how well it answers questions. Elements of the framework include circumscribing the set of questions (the "scope of understanding"), requiring general competence ("passing grade"), avoiding "ridiculous answers", but still allowing wrong and "I don't know" answers to some questions. Reaching certainty about these conditions requires exhaustive testing of the questions which is impossible for nontrivial scopes, but we show how high confidence can be achieved via random sampling and the application of probabilistic confidence bounds. We also show that accompanying answers with explanations can improve the sample complexity required to achieve acceptable bounds, because an explanation of an answer implies the ability to answer many similar questions. According to our framework, current LLMs cannot be said to understand nontrivial domains, but as the framework provides a practical recipe for testing understanding, it thus also constitutes a tool for building AI agents that do understand.