ReleaseEval: A Benchmark for Evaluating Language Models in Automated Release Note Generation

TL;DR

ReleaseEval delivers an open, reproducible benchmark for automated release-note generation with three input granularities: commit2sum, tree2sum, and diff2sum. It compiles $94{,}987$ release notes from $3{,}369$ repositories across six languages, enabling comprehensive evaluation of both fine-tuned LMs and few-shot LLMs. Empirical results show LLMs generally outperform baselines, with tree2sum receiving the strongest gains due to structured commit information, while diff2sum remains challenging because of long, fine-grained code diffs. The benchmark emphasizes reproducibility, licensing, and richer input contexts, aiming to accelerate development of robust release-note generation systems useful for developers and project stakeholders alike.

Abstract

Automated release note generation addresses the challenge of documenting frequent software updates, where manual efforts are time-consuming and prone to human error. Although recent advances in language models further enhance this process, progress remains hindered by dataset limitations, including the lack of explicit licensing and limited reproducibility, and incomplete task design that relies mainly on commit messages for summarization while overlooking fine-grained contexts such as commit hierarchies and code changes. To fill this gap, we introduce ReleaseEval, a reproducible and openly licensed benchmark designed to systematically evaluate language models for automated release note generation. ReleaseEval comprises 94,987 release notes from 3,369 repositories across 6 programming languages, and supports three task settings with three levels of input granularity: (1) commit2sum, which generates release notes from commit messages; (2) tree2sum, which incorporates commit tree structures; and (3) diff2sum, which leverages fine-grained code diffs. Both automated and human evaluations show that large language models consistently outperform traditional baselines across all tasks, achieving substantial gains on tree2sum, while still struggling on diff2sum. These findings highlight LLMs' proficiency in leveraging structured information while revealing challenges in abstracting from long code diffs.

Figures (4)

• Figure 1: The details of commits from Amazon Payments Magento 2 plugin. Figure (a): a release pipeline which includes branch-level commits. Figure (b): commit tree, which includes commit messages and their structures. Figure (c): two fine-grained code diffs, corresponding to the messages. Note: (a) and (b) highlight matched commits: blue denotes commits added on the change branch, orange indicates version-bump commits on the main branch. In (c), red marks the commits corresponding to (b), green highlights added code, and pink marks deleted code.
• Figure 2: The overall framework of ReleaseEval: (1) selecting projects from the CommitChronicle dataset eliseeva2023commit_chronicle; (2) collecting raw data of releases, commits, and code diffs based on the selected projects; (3) preprocessing the data using filtering strategies; (4) generating dataset statistics; (5) verifying dataset quality through expert evaluation; (6) splitting the dataset for training and evaluation; (7) conducting experiments; and (8) analyzing and discussing the results.
• Figure 3: Programming Language Distribution of ReleaseEval.
• Figure 4: Few‐shot performance (%) of LLMs at different scales on the commit2sum and tree2sum tasks across zero-, two-, and four-shot settings.