Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Models

TL;DR

The paper presents EWoK, a cognition-inspired framework for evaluating how language models understand basic world knowledge through controlled, context-sensitive items that test concept knowledge in minimal-pair contexts. It introduces EWoK-core-1.0, a large, template-driven dataset spanning 11 domains, and compares 20 open-weight LLMs against human performance, revealing consistent domain-dependent gaps—social knowledge is easiest, physical/spatial knowledge hardest. The authors show that log-probability-based evaluation outperforms prompting across models and discuss the implications for targeted diagnostics, interpretability, and world-model development. They also outline release safeguards and future directions for expanding the framework to multi-language settings and more sophisticated world-model assessments.

Abstract

The ability to build and reason about models of the world is essential for situated language understanding. But evaluating world modeling capabilities in modern AI systems -- especially those based on language models -- has proven challenging, in large part because of the difficulty of disentangling conceptual knowledge about the world from knowledge of surface co-occurrence statistics. This paper presents Elements of World Knowledge (EWoK), a framework for evaluating language models' understanding of the conceptual knowledge underlying world modeling. EWoK targets specific concepts from multiple knowledge domains known to be important for world modeling in humans, from social interactions (help, deceive) to spatial relations (left, right). Objects, agents, and locations in the items can be flexibly filled in, enabling easy generation of multiple controlled datasets. We then introduce EWoK-core-1.0, a dataset of 4,374 items covering 11 world knowledge domains. We evaluate 20 open-weights large language models (1.3B--70B parameters) and compare them with human performance. All tested models perform worse than humans, with results varying drastically across domains. Performance on social interactions and social properties was highest and performance on physical relations and spatial relations was lowest. Overall, this dataset highlights simple cases where even large models struggle and presents rich avenues for targeted research on LLM world modeling capabilities.

Figures (6)

• Figure 1: EWoK design, illustrated with examples from social interactions & spatial relations. Each domain contains a set of concepts, contexts, and targets. These combine to form many templates, which specify minimal pairs of contexts ($C$) and targets ($T$), such that $T_1$ matches $C_1$ but not $C_2$, and $T_2$ matches $C_2$ but not $C_1$. Each template can be combined with fillers to generate an even larger collection of items.
• Figure 2: Types of minimal pair contrasts in Context and Target pairs. Examples shown here are for one domain and concept pair. Templates may be direct, testing concepts explicitly, or indirect, testing concepts implicitly using likely scenarios (e.g., a student is more likely to talk to a teacher's colleagues rather than parents). Context and target contrasts reflect how concepts are tested. For instance, antonym contrasts words with opposing meanings, negation leverages "not", and variable swap exploits relative ordering of entities.
• Figure 3: LLM performance across world knowledge domains (evaluated with LogProbs). Here and elsewhere, the dotted line at $0.5$ denotes chance accuracy. Each dot reflects performance on a single version of EWoK-core-1.0 (see Sec. \ref{['sec:framework']} "Templates & fillers" and \ref{['sec:dataset-generation']}), with the bar reflecting the mean across the $5$ versions. LLM performance varies drastically by domain and is often substantially worse than human performance. In general, individual LLMs show similar performance patterns across domains, but these patterns are not always consistent with the human pattern.
• Figure 4: Top: LLM and human performance across target contrast (A) context contrast (B) and context type (C), evaluated with LogProbs. For examples of manipulations in A-C, see Figure \ref{['fig:contrasts']}. Dark gray line shows average model performance. Bottom: correlation between LLM accuracy and surface-level item features: (D) average item length and (E) average word frequency in the item. Humans are not sensitive or only weakly sensitive to these features, whereas model performance strongly correlates with them. The (counterintuitive) negative relationship between accuracy and word frequency is driven by the fact that hard domains happen to have high word frequency and is reversed once domain is controlled for (Table S\ref{['tab:results-mixedmodel']}).
• Figure 5: LLM performance assessed with LogProbs vs. two prompt-based tasks, Likert and Choice. Prompting is $2$-shot, and the outputs are constrained to the set of allowed values ($1$-$5$ for Likert, $1$ or $2$ for Choice); this setup was chosen to maximize model performance. Still, LogProbs is a better strategy in nearly all cases. In prompting tasks, it was common for models to generate the same value, e.g., "1" in response to any item. Our metric was designed such that even in this scenario, the 50% baseline would remain intact (see Section \ref{['sec:framework']}). Looking at Likert, without this safeguard and requiring strict inequality, the top performing Meta-LLama-3-70B model drops to $59\%$. MPT-7B, which performs comparably with LogProbs, drops to $2\%$ (See Table \ref{['tab:results-likert-nohalf']}).