Precision in Practice: Knowledge Guided Code Summarizing Grounded in Industrial Expectations

TL;DR

The paper tackles the gap between automated code summaries and industrial documentation needs by identifying three developer expectations for industrial contexts and proposing ExpSum, a four-phase, knowledge-grounded summarization framework. ExpSum combines function modeling into structured metadata, metadata quality filtering, context-aware domain term retrieval from a knowledge base, and constraint-driven two-stage prompting to generate summaries that align with domain terminology and function categorization while avoiding unnecessary details. It constructs HMSum-12 and HMSum-13 industrial benchmarks from HarmonyOS and demonstrates substantial improvements over baselines on these datasets, with up to $26.71\%$ gains in $BLEU$-4 and $20.10\%$ in $ROUGE$-L, and shows strong cross-version and cross-project generalizability via open-source benchmarks and LLM-based human judgments. The work highlights the practical value of integrating domain knowledge and explicit constraints into LLM-driven code summarization for industrial software documentation and maintenance.

Abstract

Code summaries are essential for helping developers understand code functionality and reducing maintenance and collaboration costs. Although recent advances in large language models (LLMs) have significantly improved automatic code summarization, the practical usefulness of generated summaries in industrial settings remains insufficiently explored. In collaboration with documentation experts from the industrial HarmonyOS project, we conducted a questionnaire study showing that over 57.4% of code summaries produced by state-of-the-art approaches were rejected due to violations of developers' expectations for industrial documentation. Beyond semantic similarity to reference summaries, developers emphasize additional requirements, including the use of appropriate domain terminology, explicit function categorization, and the avoidance of redundant implementation details. To address these expectations, we propose ExpSum, an expectation-aware code summarization approach that integrates function metadata abstraction, informative metadata filtering, context-aware domain knowledge retrieval, and constraint-driven prompting to guide LLMs in generating structured, expectation-aligned summaries. We evaluate ExpSum on the HarmonyOS project and widely used code summarization benchmarks. Experimental results show that ExpSum consistently outperforms all baselines, achieving improvements of up to 26.71% in BLEU-4 and 20.10% in ROUGE-L on HarmonyOS. Furthermore, LLM-based evaluations indicate that ExpSum-generated summaries better align with developer expectations across other projects, demonstrating its effectiveness for industrial code documentation.

Figures (2)

• Figure 1: Overview of ExpSum. It first extracts package-level documentation and domain terms from projects to construct a domain knowledge base that preserves the contextual semantics of terms. Before generating function-level summaries with LLMs, each function is modeled into a set of metadata, whose content informativeness is then evaluated. Then, a context-aware knowledge retrieval algorithm retrieves domain terms semantically related to the function model embeddings. The retrieved terms are given to the LLM and combined with constraint-based prompting techniques to guide the LLM in code summary generation.
• Figure 2: Prompt templates used for the two stages, respectively. The "Knowledge Entries" in Stage 1 are assembled based on the retrieved terms from the knowledge base, and the "<Input Metadata Set>" in both Stages 1 and 2 are subjected to metadata information checking to enhance the usability.