Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Models Know Models Best: Evaluation via Model-Preferred Formats

Joonhak Lee, Sungmok Jung, Jongyeon Park, Jaejin Lee

TL;DR

This work investigates how MCQA evaluation formats—symbol-based and cloze-style—shape LLM performance in a task-dependent manner. It demonstrates that format mismatches arise from underlying task structure and model inductive biases, across multiple decoder-based LLMs and benchmarks. The authors propose a dynamic format-alignment approach using a lightweight classifier trained on either human-annotated or model-generated signals to select the optimal format per instance, achieving substantial zero-shot gains, especially on completion-oriented tasks, and generalizing across models via majority voting. The results advocate moving beyond fixed-format MCQA evaluation to format-aware assessments that better reveal latent model capabilities and guide robust evaluation practices.

Abstract

Performance of Large Language Models (LLMs) on multiple-choice tasks differs markedly between symbol-based and cloze-style evaluation formats. The observed discrepancies are systematically attributable to task characteristics: natural language continuation benefits from likelihood scoring, whereas explicit comparison is better suited to symbol-based selection. These trends are consistent across various decoder-based LLMs, indicating model-agnostic effects. To address these inconsistencies, a dynamic format-alignment strategy is introduced that employs a lightweight classifier trained on latent model-preference signals. In contrast to human-designed heuristics, which often degrade performance, this approach uses model-generated signals to determine the optimal format for each problem instance. The proposed method achieves substantial and consistent improvements in zero-shot accuracy across reasoning and knowledge benchmarks, better revealing the models' latent capabilities.

Models Know Models Best: Evaluation via Model-Preferred Formats

TL;DR

This work investigates how MCQA evaluation formats—symbol-based and cloze-style—shape LLM performance in a task-dependent manner. It demonstrates that format mismatches arise from underlying task structure and model inductive biases, across multiple decoder-based LLMs and benchmarks. The authors propose a dynamic format-alignment approach using a lightweight classifier trained on either human-annotated or model-generated signals to select the optimal format per instance, achieving substantial zero-shot gains, especially on completion-oriented tasks, and generalizing across models via majority voting. The results advocate moving beyond fixed-format MCQA evaluation to format-aware assessments that better reveal latent model capabilities and guide robust evaluation practices.

Abstract

Performance of Large Language Models (LLMs) on multiple-choice tasks differs markedly between symbol-based and cloze-style evaluation formats. The observed discrepancies are systematically attributable to task characteristics: natural language continuation benefits from likelihood scoring, whereas explicit comparison is better suited to symbol-based selection. These trends are consistent across various decoder-based LLMs, indicating model-agnostic effects. To address these inconsistencies, a dynamic format-alignment strategy is introduced that employs a lightweight classifier trained on latent model-preference signals. In contrast to human-designed heuristics, which often degrade performance, this approach uses model-generated signals to determine the optimal format for each problem instance. The proposed method achieves substantial and consistent improvements in zero-shot accuracy across reasoning and knowledge benchmarks, better revealing the models' latent capabilities.
Paper Structure (30 sections, 3 figures, 32 tables, 1 algorithm)

This paper contains 30 sections, 3 figures, 32 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. MCQA Evaluation Formats: Symbol and Cloze
  4. Sensitivity to MCQA Evaluation Formats
  5. Symbol vs. Cloze: A Baseline Study
  6. Evaluation Setup
  7. Models.
  8. Benchmarks.
  9. Symbol-based evaluation format.
  10. Cloze-style evaluation format.
  11. Prompt construction.
  12. Few-shot evaluation.
  13. Findings
  14. Tasks have evaluation format preferences.
  15. Effect of in-context learning.
  16. ...and 15 more sections

Figures (3)

  • Figure 1: Examples of prompts and candidates used for log-likelihood scoring in MMLU and HellaSwag under symbol and cloze evaluation formats. Correct answers are highlighted in red.
  • Figure 2: Effect of in-context demonstrations on format mismatch. The vertical axis shows the performance difference between the symbol and cloze formats (symbol minus cloze, in percentage points), while the horizontal axis indicates the number of in-context demonstration examples.
  • Figure 3: Overview of instance-wise evaluation format selection. The classifier predicts the preferred format (symbol or cloze) for each instance. Based on the selected format, both the prompt fed into the LLM and the candidate set used for scoring are determined.