SmartNote: An LLM-Powered, Personalised Release Note Generator That Just Works
Farbod Daneshyan, Runzhi He, Jianyu Wu, Minghui Zhou
TL;DR
Release notes are essential but time-consuming to produce, and existing tools struggle with audience-specific personalization and workflow integration. SmartNote introduces a five-stage LLM-powered pipeline that aggregates changes from commits and PRs, infers project domain and audience needs, and personalises release notes without imposing strict conventions. It combines an Info Retriever, Settings Generator, Commit Analyser, Change Summariser, and RN Composer, supported by prompt engineering and a domain-aware classifier, achieving high completeness and strong organisational structure across 23 open-source projects, with a reported commit coverage of $81\%$ and superior performance on several quality metrics. The work demonstrates that prompt engineering and context-aware generation enable broadly applicable, high-quality release notes, offering a practical path to scalable, personalized documentation in open-source software, with public data and code available for replication and extension. $RN_{final} = format(sort extunderscore filter(RN, ST, MST), WS)$ illustrates the core output formulation linking the generated changes to the final release note structure.
Abstract
The release note is a crucial document outlining changes in new software versions. Yet, many developers view the process of writing software release notes as a tedious and dreadful task. Consequently, numerous tools have been developed by researchers and practitioners to automate the generation of software release notes. However, these tools fail to consider project domain and target audience for personalisation, limiting their relevance and conciseness. Additionally, they suffer from limited applicability, often necessitating significant workflow adjustments and adoption efforts, hindering practical use and stressing developers. Despite recent advancements in natural language processing and the proven capabilities of large language models in various code and text-related tasks, there are no existing studies investigating the integration and utilisation of LLMs in automated release note generation. Therefore, we propose SmartNote, a novel and widely applicable release note generation approach that produces high-quality, contextually personalised release notes using LLM technology. SmartNote aggregates changes and uses an LLM to describe and summarise the changes using code, commit, and pull request details. It categorises and scores commits to generate structured and concise release notes of prioritised changes. Our human and automatic evaluations reveal that SmartNote outperforms or achieves comparable performance to DeepRelease, Conventional Changelog, and the projects'original release notes across four quality metrics: completeness, clarity, conciseness, and organisation. In both evaluations, SmartNote ranked first for completeness and organisation, while clarity ranked first in the human evaluation. A further evaluation demonstrates that SmartNote is effective in terms of context awareness and applicability.