CiteBench: A benchmark for Scientific Citation Text Generation

TL;DR

Citations anchor scientific arguments and are increasingly hard to track amid rapid publication. CiteBench unifies four divergent citation-text task designs into a single, general framework and provides baselines and a standardized evaluation kit. The study reveals that simple extractive baselines can be strong, transfer learning can improve cross-domain performance, and discourse-aware metrics illuminate qualitative differences between generated and gold citations. It also highlights replicability concerns and calls for richer inputs, task refinements, and broader domain coverage.

Abstract

Science progresses by building upon the prior body of knowledge documented in scientific publications. The acceleration of research makes it hard to stay up-to-date with the recent developments and to summarize the ever-growing body of prior work. To address this, the task of citation text generation aims to produce accurate textual summaries given a set of papers-to-cite and the citing paper context. Due to otherwise rare explicit anchoring of cited documents in the citing paper, citation text generation provides an excellent opportunity to study how humans aggregate and synthesize textual knowledge from sources. Yet, existing studies are based upon widely diverging task definitions, which makes it hard to study this task systematically. To address this challenge, we propose CiteBench: a benchmark for citation text generation that unifies multiple diverse datasets and enables standardized evaluation of citation text generation models across task designs and domains. Using the new benchmark, we investigate the performance of multiple strong baselines, test their transferability between the datasets, and deliver new insights into the task definition and evaluation to guide future research in citation text generation. We make the code for CiteBench publicly available at https://github.com/UKPLab/citebench.

Figures (4)

• Figure 1: Citation text generation task, text in the grey box adopted from devlin-etal-2019-bert.
• Figure 2: Data examples extracted from the datasets (left). <abs> -- citing paper's abstract, <ctx_b> -- context before target citation text, <ctx_a> -- context after target citation text, rightmost column -- generated citation text. Title in ABURAED marked with orange.
• Figure 3: Citation intent distribution (left) for model outputs (top) and datasets (bottom), and KL divergence between datasets and model outputs (right).
• Figure 4: CORWA tag distribution (left) for model outputs (top) and datasets (bottom), and KL divergence between datasets and model outputs (right). Empty cells in XING (right) denote $\infty$ due to missing labels in system predictions. Note the scale differences between the KL divergence plots here and in Figure \ref{['fig:acl-arc']}, kept for presentation clarity.