Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dialogue Games for Benchmarking Language Understanding: Motivation, Taxonomy, Strategy

David Schlangen

TL;DR

The paper argues that Situated Language Understanding (SLU) cannot be adequately measured with traditional, static NLU benchmarks. It introduces Dialogue Games as principled, task-oriented benchmarks that embed language use in goal-directed interactions, grounded in a formal model of SLU capabilities. A detailed taxonomy and evaluation toolkit are proposed, linking game design to specific cognitive processes and enabling multiple data-generation and probing modalities. The approach aims to provide construct-valid, ecologically meaningful assessment and a scalable path from simple to complex SLU tasks, with attention to safety and ethical implications in embodied systems.

Abstract

How does one measure "ability to understand language"? If it is a person's ability that is being measured, this is a question that almost never poses itself in an unqualified manner: Whatever formal test is applied, it takes place on the background of the person's language use in daily social practice, and what is measured is a specialised variety of language understanding (e.g., of a second language; or of written, technical language). Computer programs do not have this background. What does that mean for the applicability of formal tests of language understanding? I argue that such tests need to be complemented with tests of language use embedded in a practice, to arrive at a more comprehensive evaluation of "artificial language understanding". To do such tests systematically, I propose to use "Dialogue Games" -- constructed activities that provide a situational embedding for language use. I describe a taxonomy of Dialogue Game types, linked to a model of underlying capabilites that are tested, and thereby giving an argument for the \emph{construct validity} of the test. I close with showing how the internal structure of the taxonomy suggests an ordering from more specialised to more general situational language understanding, which potentially can provide some strategic guidance for development in this field.

Dialogue Games for Benchmarking Language Understanding: Motivation, Taxonomy, Strategy

TL;DR

The paper argues that Situated Language Understanding (SLU) cannot be adequately measured with traditional, static NLU benchmarks. It introduces Dialogue Games as principled, task-oriented benchmarks that embed language use in goal-directed interactions, grounded in a formal model of SLU capabilities. A detailed taxonomy and evaluation toolkit are proposed, linking game design to specific cognitive processes and enabling multiple data-generation and probing modalities. The approach aims to provide construct-valid, ecologically meaningful assessment and a scalable path from simple to complex SLU tasks, with attention to safety and ethical implications in embodied systems.

Abstract

How does one measure "ability to understand language"? If it is a person's ability that is being measured, this is a question that almost never poses itself in an unqualified manner: Whatever formal test is applied, it takes place on the background of the person's language use in daily social practice, and what is measured is a specialised variety of language understanding (e.g., of a second language; or of written, technical language). Computer programs do not have this background. What does that mean for the applicability of formal tests of language understanding? I argue that such tests need to be complemented with tests of language use embedded in a practice, to arrive at a more comprehensive evaluation of "artificial language understanding". To do such tests systematically, I propose to use "Dialogue Games" -- constructed activities that provide a situational embedding for language use. I describe a taxonomy of Dialogue Game types, linked to a model of underlying capabilites that are tested, and thereby giving an argument for the \emph{construct validity} of the test. I close with showing how the internal structure of the taxonomy suggests an ordering from more specialised to more general situational language understanding, which potentially can provide some strategic guidance for development in this field.
Paper Structure (18 sections, 5 figures, 1 table)

This paper contains 18 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background On Measurement
  3. Current Practices in Measuring NLU
  4. SLU is Different From NLU
  5. SLU Requires Different Benchmarking Methods: Dialogue Games
  6. The Construct: A Model of Capabilities Involved in SLU
  7. A Taxonomy of Game Types
  8. Environment
  9. Setting
  10. Game
  11. Rollout
  12. Agent/Agent Play
  13. Human/Agent Play
  14. Representation Probing
  15. Strategy: From Simple to Complex Dialogue Games
  16. ...and 3 more sections

Figures (5)

  • Figure 1: The structure of relations between the research objects model, dataset, task, environment, setting, game, and cognitive capability. Adapted from schlangen:tasks.
  • Figure 2: An Example of a GLUE-type task (from the BoolQ subset, clark-etal-2019-boolq, as cited in superGLUE)
  • Figure 3: Representational Domains (bottom) and Anchoring Processes (top) Structuring the Situated Agent
  • Figure 4: The main components of the proposed taxonomy
  • Figure 5: A partial order on the space of Dialogue Games. $\sim$ denotes "similar complexity", $>$ denotes "leading to higher complexity".