Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM Olympiad: Why Model Evaluation Needs a Sealed Exam

Jan Christian Blaise Cruz, Alham Fikri Aji

Abstract

Benchmarks and leaderboards are how NLP most often communicates progress, but in the LLM era they are increasingly easy to misread. Scores can reflect benchmark-chasing, hidden evaluation choices, or accidental exposure to test content -- not just broad capability. Closed benchmarks delay some of these issues, but reduce transparency and make it harder for the community to learn from results. We argue for a complementary practice: an Olympiad-style evaluation event where problems are sealed until evaluation, submissions are frozen in advance, and all entries run through one standardized harness. After scoring, the full task set and evaluation code are released so results can be reproduced and audited. This design aims to make strong performance harder to ``manufacture'' and easier to trust.

LLM Olympiad: Why Model Evaluation Needs a Sealed Exam

Abstract

Benchmarks and leaderboards are how NLP most often communicates progress, but in the LLM era they are increasingly easy to misread. Scores can reflect benchmark-chasing, hidden evaluation choices, or accidental exposure to test content -- not just broad capability. Closed benchmarks delay some of these issues, but reduce transparency and make it harder for the community to learn from results. We argue for a complementary practice: an Olympiad-style evaluation event where problems are sealed until evaluation, submissions are frozen in advance, and all entries run through one standardized harness. After scoring, the full task set and evaluation code are released so results can be reproduced and audited. This design aims to make strong performance harder to ``manufacture'' and easier to trust.
Paper Structure (64 sections, 12 figures)

This paper contains 64 sections, 12 figures.

Table of Contents

  1. Introduction
  2. Premise.
  3. Intuition.
  4. Our position.
  5. Roadmap.
  6. The Limits of Today's Benchmarking Toolbox
  7. Open benchmarks
  8. Closed benchmarks
  9. Shared tasks
  10. Takeaway
  11. The LLM Olympiad: A Sealed Evaluation Protocol
  12. Expectations
  13. Design principles
  14. Why it is needed
  15. Fragility.
  16. ...and 49 more sections

Figures (12)

  • Figure 1: Comparison of evaluation formats across three dimensions. Open benchmarks are fully transparent but expose tasks to optimization and leakage. Shared tasks hide test labels but evaluate against a known target under non-standardized protocols. The proposed Olympiad format seals tasks until evaluation, enforces a standardized harness, and releases all artifacts afterward — combining the strengths of existing formats while mitigating their core weaknesses.
  • Figure 2: End-to-end flow of the proposed LLM Olympiad. Task authors propose problems via an open call; organizers curate and seal the task bundle before any model submission is evaluated. Model and system submitters prepare broadly without knowledge of task content and freeze their submissions in advance. Evaluation is run centrally under a standardized harness, with scores released to submitters and all artifacts — tasks, scoring code, and harness — released publicly afterward for community audit and reuse.
  • Figure 3: Illustrative excerpt of a contest syllabus. The event preserves surprise in task content while making rules and constraints predictable.
  • Figure 4: A lightweight consistency probe can detect instability or endpoint drift without changing the main scoring rules.
  • Figure 5: Illustrative run manifest fields. Even minimal manifests improve auditability and reduce ambiguity about evaluation conditions.
  • ...and 7 more figures