Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembench

David Schlangen, Sherzod Hakimov, Chalamalasetti Kranti, Jonathan Jordan, Philipp Sadler

Abstract

There are currently two main paradigms for evaluating large language models (LLMs), reference-based evaluation and preference-based evaluation. The first, carried over from the evaluation of machine learning models in general, relies on pre-defined task instances, for which reference task executions are available. The second, best exemplified by the LM-arena, relies on (often self-selected) users bringing their own intents to a site that routes these to several models in parallel, among whose responses the user then selects their most preferred one. The former paradigm hence excels at control over what is tested, while the latter comes with higher ecological validity, testing actual use cases interactively. Recently, a third complementary paradigm has emerged that combines some of the strengths of these approaches, offering control over multi-turn, reference-free, repeatable interactions, while stressing goal-directedness: dialogue game based evaluation. While the utility of this approach has been shown by several projects, its adoption has been held back by the lack of a mature, easily re-usable implementation. In this paper, we present clembench, which has been in continuous development since 2023 and has in its latest release been optimized for ease of general use. We describe how it can be used to benchmark one's own models (using a provided set of benchmark game instances in English), as well as how easily the benchmark itself can be extended with new, tailor-made targeted tests.

A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembench

Abstract

There are currently two main paradigms for evaluating large language models (LLMs), reference-based evaluation and preference-based evaluation. The first, carried over from the evaluation of machine learning models in general, relies on pre-defined task instances, for which reference task executions are available. The second, best exemplified by the LM-arena, relies on (often self-selected) users bringing their own intents to a site that routes these to several models in parallel, among whose responses the user then selects their most preferred one. The former paradigm hence excels at control over what is tested, while the latter comes with higher ecological validity, testing actual use cases interactively. Recently, a third complementary paradigm has emerged that combines some of the strengths of these approaches, offering control over multi-turn, reference-free, repeatable interactions, while stressing goal-directedness: dialogue game based evaluation. While the utility of this approach has been shown by several projects, its adoption has been held back by the lack of a mature, easily re-usable implementation. In this paper, we present clembench, which has been in continuous development since 2023 and has in its latest release been optimized for ease of general use. We describe how it can be used to benchmark one's own models (using a provided set of benchmark game instances in English), as well as how easily the benchmark itself can be extended with new, tailor-made targeted tests.

Paper Structure

This paper contains 20 sections, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Evaluation Through Self-Play of Dialogue Games
  3. The General Idea
  4. What makes games like these useful evaluation instruments?
  5. How can such games be played with LLMs?
  6. Some Unique Properties
  7. Existing Frameworks
  8. clemcore / clembench
  9. Running clembench
  10. Extending the Benchmark
  11. Creating New Instances
  12. Creating New Games
  13. Multilingual Evaluation with clembench: Adding New Languages
  14. "Vibe-checking" the Agentic Abilities of a Model
  15. Evaluating the Evaluator
  16. ...and 5 more sections

Figures (6)

  • Figure 1: A constructed example of a successful episode of the two-player Dialogue Game Taboo, as played by humans
  • Figure 2: Llama-3.1-405B in prompted self-play of the same instance of Taboo as in Figure\ref{['fig:taboo-hh']} (excerpt).
  • Figure 3: Schematic view of the abstraction layers provided by clemcore / clembench
  • Figure 4: A minimal model registry entry for a model hosted on a public repository and inferenced with the transformers library
  • Figure 5: Development of achieved scores on clembench since 2023. Interactive version available at https://clembench.github.io/leaderboard.html ("trends").
  • ...and 1 more figures