Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exposing Assumptions in AI Benchmarks through Cognitive Modelling

Jonathan H. Rystrøm, Kenneth C. Enevoldsen

TL;DR

The paper addresses the problem that AI benchmarks embed implicit, often vague assumptions about measured constructs. It introduces a cognitive modelling approach using Structural Equation Modeling (SEM) to explicitly operationalize latent constructs (e.g., language ability $G$, cultural knowledge $CK$, and alignment $HHH$) and to connect them to benchmark data across languages, with cross-lingual alignment transfer as the exemplar. It contributes a framework for exposing assumptions, guiding dataset development, and enabling rigorous, cumulative evaluation of Generative AI traits, while acknowledging limits in construct definition and data requirements. The work aims to enhance transparency and construct validity in multilingual AI evaluation, facilitating more robust cross-cultural benchmarking and theoretical grounding for LLM capabilities.

Abstract

Cultural AI benchmarks often rely on implicit assumptions about measured constructs, leading to vague formulations with poor validity and unclear interrelations. We propose exposing these assumptions using explicit cognitive models formulated as Structural Equation Models. Using cross-lingual alignment transfer as an example, we show how this approach can answer key research questions and identify missing datasets. This framework grounds benchmark construction theoretically and guides dataset development to improve construct measurement. By embracing transparency, we move towards more rigorous, cumulative AI evaluation science, challenging researchers to critically examine their assessment foundations.

Exposing Assumptions in AI Benchmarks through Cognitive Modelling

TL;DR

The paper addresses the problem that AI benchmarks embed implicit, often vague assumptions about measured constructs. It introduces a cognitive modelling approach using Structural Equation Modeling (SEM) to explicitly operationalize latent constructs (e.g., language ability , cultural knowledge , and alignment ) and to connect them to benchmark data across languages, with cross-lingual alignment transfer as the exemplar. It contributes a framework for exposing assumptions, guiding dataset development, and enabling rigorous, cumulative evaluation of Generative AI traits, while acknowledging limits in construct definition and data requirements. The work aims to enhance transparency and construct validity in multilingual AI evaluation, facilitating more robust cross-cultural benchmarking and theoretical grounding for LLM capabilities.

Abstract

Cultural AI benchmarks often rely on implicit assumptions about measured constructs, leading to vague formulations with poor validity and unclear interrelations. We propose exposing these assumptions using explicit cognitive models formulated as Structural Equation Models. Using cross-lingual alignment transfer as an example, we show how this approach can answer key research questions and identify missing datasets. This framework grounds benchmark construction theoretically and guides dataset development to improve construct measurement. By embracing transparency, we move towards more rigorous, cumulative AI evaluation science, challenging researchers to critically examine their assessment foundations.
Paper Structure (9 sections, 2 figures, 1 table)

This paper contains 9 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. (Re)constructing Crosslingual Alignment Transfer
  3. Limitations
  4. Estimating Cross-cultural Language Ability: A Simplified Example
  5. Data
  6. Modelling and Hypothesis Testing
  7. Results
  8. Learnings from the Analysis
  9. Benchmark Descriptions

Figures (2)

  • Figure 1: Simplified Structural Equation Model: Cross-lingual Alignment Transfer. Latent variables (circles): Language ability ($G$), Cultural knowledge ($CK$), Alignment ($HHH$) for English and Danish. Observable variables (rectangles): Language-specific tasks (e.g., CoLA, SocialIQA). Arrows: single-headed for hypothesised relationships, bidirectional for covariance between latent variables.
  • Figure 2: Caption