Integrating Code Metrics into Automated Documentation Generation for Computational Notebooks

TL;DR

The paper tackles automated code documentation generation for computational notebooks and the underutilization of source-code metrics as auxiliary cues. It introduces a two-stage dataset creation workflow from a CodeSearchNet‑inspired corpus and evaluates a CNN‑RNN baseline alongside a GPT‑3.5 few‑shot model, with and without metric augmentation. Empirical results show that code metrics provide structural context that improves generation quality, achieving improvements such as $BLEU\text{-}1$ up by about $6\%$, $ROUGE\text{-}L$ F1 up by about $3\%$, and $BERTScore$ up by about $9\%$ in respective setups. The findings support the utility of metric‑informed conditioning for notebook documentation and point to scalable directions for broader software‑engineering documentation tasks.

Abstract

Effective code documentation is essential for collaboration, comprehension, and long-term software maintainability, yet developers often neglect it due to its repetitive nature. Automated documentation generation has evolved from heuristic and rule-based methods to neural network-based and large language model (LLM)-based approaches. However, existing methods often overlook structural and quantitative characteristics of code that influence readability and comprehension. Prior research suggests that code metrics capture information relevant to program understanding. Building on these insights, this paper investigates the role of source code metrics as auxiliary signals for automated documentation generation, focusing on computational notebooks, a popular medium among data scientists that integrates code, narrative, and results but suffers from inconsistent documentation. We propose a two-stage approach. First, the CodeSearchNet dataset construction process was refined to create a specialized dataset from over 17 million code and markdown cells. After structural and semantic filtering, approximately 36,734 high-quality (code, markdown) pairs were extracted. Second, two modeling paradigms, a lightweight CNN-RNN architecture and a few-shot GPT-3.5 architecture, were evaluated with and without metric information. Results show that incorporating code metrics improves the accuracy and contextual relevance of generated documentation, yielding gains of 6% in BLEU-1 and 3% in ROUGE-L F1 for CNN-RNN-based architecture, and 9% in BERTScore F1 for LLM-based architecture. These findings demonstrate that integrating code metrics provides valuable structural context, enhancing automated documentation generation across diverse model families.

Figures (5)

• Figure 1: The Dataset Construction Methodology
• Figure 2: Model Comparison for Code and Documentation Similarity in the Ground Truth Dataset
• Figure 3: The CNN-RNN Architecture for Documentation Generation
• Figure 4: The Few-shot LLM-based Architecture for Documentation Generation
• Figure 5: Overall Evaluation Results of the GPT‑3.5 Model Across BLEU, ROUGE, and BERTScore Metrics.