Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge

Karen Zhou, Chenhao Tan

TL;DR

AutoChecklist, an open-source library that unifies checklist-based evaluation into composable pipelines, and a case study on ICLR peer review rebuttals demonstrates flexible domain adaptation.

Abstract

Checklists have emerged as a popular approach for interpretable and fine-grained evaluation, particularly with LLM-as-a-Judge. Beyond evaluation, these structured criteria can serve as signals for model alignment, reinforcement learning, and self-correction. To support these use cases, we present AutoChecklist, an open-source library that unifies checklist-based evaluation into composable pipelines. At its core is a taxonomy of five checklist generation abstractions, each encoding a distinct strategy for deriving evaluation criteria. A modular Generator $\rightarrow$ Refiner $\rightarrow$ Scorer pipeline connects any generator with a unified scorer, and new configurations can be registered via prompt templates alone. The library ships with ten built-in pipelines implementing published approaches and supports multiple LLM providers (OpenAI, OpenRouter, vLLM). Beyond the Python API, the library includes a CLI for off-the-shelf evaluation and a web interface for interactive exploration. Validation experiments confirm that these checklist methods significantly align with human preferences and quality ratings, and a case study on ICLR peer review rebuttals demonstrates flexible domain adaptation. AutoChecklist is publicly available at https://github.com/ChicagoHAI/AutoChecklist.

AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge

TL;DR

AutoChecklist, an open-source library that unifies checklist-based evaluation into composable pipelines, and a case study on ICLR peer review rebuttals demonstrates flexible domain adaptation.

Abstract

Checklists have emerged as a popular approach for interpretable and fine-grained evaluation, particularly with LLM-as-a-Judge. Beyond evaluation, these structured criteria can serve as signals for model alignment, reinforcement learning, and self-correction. To support these use cases, we present AutoChecklist, an open-source library that unifies checklist-based evaluation into composable pipelines. At its core is a taxonomy of five checklist generation abstractions, each encoding a distinct strategy for deriving evaluation criteria. A modular Generator Refiner Scorer pipeline connects any generator with a unified scorer, and new configurations can be registered via prompt templates alone. The library ships with ten built-in pipelines implementing published approaches and supports multiple LLM providers (OpenAI, OpenRouter, vLLM). Beyond the Python API, the library includes a CLI for off-the-shelf evaluation and a web interface for interactive exploration. Validation experiments confirm that these checklist methods significantly align with human preferences and quality ratings, and a case study on ICLR peer review rebuttals demonstrates flexible domain adaptation. AutoChecklist is publicly available at https://github.com/ChicagoHAI/AutoChecklist.
Paper Structure (9 sections, 5 figures, 5 tables)

This paper contains 9 sections, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. System Details
  4. Deployment and Usage
  5. Validation
  6. Case Study: Author Rebuttal Checklists
  7. Conclusion and Future Work
  8. Additional Figures
  9. Code Samples

Figures (5)

  • Figure 1: In the Compare page of AutoChecklist's UI, users select methods and enter an input to generate side-by-side checklists, enabling direct comparison of generation strategies on the same task.
  • Figure 2: Instance-level generators produce one checklist per input; criteria are tailored to each specific task (target). Corpus-level generators produce one checklist for an entire dataset of targets, capturing general quality patterns derived from higher-level signals.
  • Figure 3: The Batch results page showing per-item score breakdowns and aggregate statistics for a dataset evaluation run.
  • Figure 4: Additional UI pages to support prompt iteration and dataset evaluation.
  • Figure 5: Additional results from validation and case studies.