Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CoverBench: A Challenging Benchmark for Complex Claim Verification

Alon Jacovi, Moran Ambar, Eyal Ben-David, Uri Shaham, Amir Feder, Mor Geva, Dror Marcus, Avi Caciularu

TL;DR

CoverBench, a challenging benchmark focused on verifying LM outputs in complex reasoning settings, is introduced, providing a diversified evaluation for complex claim verification in a variety of domains, types of reasoning, relatively long inputs, and a variety of standardizations.

Abstract

There is a growing line of research on verifying the correctness of language models' outputs. At the same time, LMs are being used to tackle complex queries that require reasoning. We introduce CoverBench, a challenging benchmark focused on verifying LM outputs in complex reasoning settings. Datasets that can be used for this purpose are often designed for other complex reasoning tasks (e.g., QA) targeting specific use-cases (e.g., financial tables), requiring transformations, negative sampling and selection of hard examples to collect such a benchmark. CoverBench provides a diversified evaluation for complex claim verification in a variety of domains, types of reasoning, relatively long inputs, and a variety of standardizations, such as multiple representations for tables where available, and a consistent schema. We manually vet the data for quality to ensure low levels of label noise. Finally, we report a variety of competitive baseline results to show CoverBench is challenging and has very significant headroom. The data is available at https://huggingface.co/datasets/google/coverbench .

CoverBench: A Challenging Benchmark for Complex Claim Verification

TL;DR

CoverBench, a challenging benchmark focused on verifying LM outputs in complex reasoning settings, is introduced, providing a diversified evaluation for complex claim verification in a variety of domains, types of reasoning, relatively long inputs, and a variety of standardizations.

Abstract

There is a growing line of research on verifying the correctness of language models' outputs. At the same time, LMs are being used to tackle complex queries that require reasoning. We introduce CoverBench, a challenging benchmark focused on verifying LM outputs in complex reasoning settings. Datasets that can be used for this purpose are often designed for other complex reasoning tasks (e.g., QA) targeting specific use-cases (e.g., financial tables), requiring transformations, negative sampling and selection of hard examples to collect such a benchmark. CoverBench provides a diversified evaluation for complex claim verification in a variety of domains, types of reasoning, relatively long inputs, and a variety of standardizations, such as multiple representations for tables where available, and a consistent schema. We manually vet the data for quality to ensure low levels of label noise. Finally, we report a variety of competitive baseline results to show CoverBench is challenging and has very significant headroom. The data is available at https://huggingface.co/datasets/google/coverbench .
Paper Structure (39 sections, 3 figures, 5 tables)

This paper contains 39 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Benchmark Scope
  3. Domains.
  4. Sources of complexity.
  5. Datasets.
  6. Constructing CoverBench
  7. Conversion to the Schema
  8. Schema.
  9. QA pairs to declarative statements.
  10. Representing structure.
  11. Sampling Negative Examples
  12. Example Selection
  13. Vetting
  14. The CoverBench Benchmark
  15. Experimental Setup and Results
  16. ...and 24 more sections

Figures (3)

  • Figure 1: CoverBench contains true and false claims that require implicit complex reasoning to verify in a variety of domains and settings.
  • Figure 2: Distribution of the source datasets and the text domains in CoverBench.
  • Figure 3: Distribution of the sources of complexity in CoverBench. Long context in this figure refers to examples with over 3,000 tokens with the Mixtral-8x7b-Instruct tokenizer.