APIDocBooster: An Extract-Then-Abstract Framework Leveraging Large Language Models for Augmenting API Documentation

TL;DR

APIDocBooster tackles the problem of enhancing API documentation by combining external information with structured, faithful summaries. It introduces a two-stage framework (CSSC and UPSUM) that first extracts section-aware insights from multiple sources and then verbosely abstracts them under GPT-4 guidance, with an intermediate extractive update step to reduce hallucinations. A novel dataset, APISumBench, enables automatic evaluation of both the sentence-level classification and extractive-update summarization components, and human studies show improvements in informativeness, relevance, and faithfulness over GPT-4 alone. The approach achieves large-margin gains across automatic metrics and human judgments, suggesting practical impact for maintaining and augmenting API docs with diverse, community-driven knowledge while maintaining provenance.

Abstract

API documentation is often the most trusted resource for programming. Many approaches have been proposed to augment API documentation by summarizing complementary information from external resources such as Stack Overflow. Existing extractive-based summarization approaches excel in producing faithful summaries that accurately represent the source content without input length restrictions. Nevertheless, they suffer from inherent readability limitations. On the other hand, our empirical study on the abstractive-based summarization method, i.e., GPT-4, reveals that GPT-4 can generate coherent and concise summaries but presents limitations in terms of informativeness and faithfulness. We introduce APIDocBooster, an extract-then-abstract framework that seamlessly fuses the advantages of both extractive (i.e., enabling faithful summaries without length limitation) and abstractive summarization (i.e., producing coherent and concise summaries). APIDocBooster consists of two stages: (1) \textbf{C}ontext-aware \textbf{S}entence \textbf{S}ection \textbf{C}lassification (CSSC) and (2) \textbf{UP}date \textbf{SUM}marization (UPSUM). CSSC classifies API-relevant information collected from multiple sources into API documentation sections. UPSUM first generates extractive summaries distinct from the original API documentation and then generates abstractive summaries guided by extractive summaries through in-context learning. To enable automatic evaluation of APIDocBooster, we construct the first dataset for API document augmentation. Our automatic evaluation results reveal that each stage in APIDocBooster outperforms its baselines by a large margin. Our human evaluation also demonstrates the superiority of APIDocBooster over GPT-4 and shows that it improves informativeness, relevance, and faithfulness by 13.89\%, 15.15\%, and 30.56\%, respectively.

Figures (2)

• Figure 1: Overview of APIDocBooster. (1) we train a context identifier and a sentence section classifier to identify the context of each sentence and divide each sentence into either one of the three sections or API-irrelevant, respectively. (2) we generate extractive summaries for each section by using our proposed update summarization algorithm. We then generate abstractive summaries guided by extractive summaries.
• Figure 2: Example of APIDocBooster's summarization process. (1) Given sentences extracted from external sources, we train a sentence section classifier to classify sentences into either one of three API documentation sections (i.e., functionality, parameter, notes) or API-irrelevant one. Then we leverage our extractive update summarization algorithm to generate extractive summaries for each section. Finally, we ask a large language model to generate abstractive summaries for each section guided by extractive summaries.