Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DocuMint: Docstring Generation for Python using Small Language Models

Bibek Poudel, Adam Cook, Sekou Traore, Shelah Ameli

TL;DR

DocuMint addresses the gap in evaluating docstring quality generated by small language models by proposing a dedicated evaluation framework and a large supervised dataset of 100,000 Python function-docstring pairs. The study benchmarks multiple CodeLLMs on three metrics—accuracy, conciseness, and clarity—using both automated scores (notably with BERTScore and readability measures) and human judgments, finding that Llama3 8B performs strongest on quantitative metrics while CodeGemma 7B shines in human evaluations. A LoRA-based fine-tuning of CodeGemma 2B using the DocuMint data yields substantial improvements in conciseness and overall docstring quality, demonstrating the value of dataset-driven fine-tuning. The authors release the DocuMint dataset, the fine-tuned CodeGemma 2B model, and the associated code to support further research and practical adoption in software documentation tooling.

Abstract

Effective communication, specifically through documentation, is the beating heart of collaboration among contributors in software development. Recent advancements in language models (LMs) have enabled the introduction of a new type of actor in that ecosystem: LM-powered assistants capable of code generation, optimization, and maintenance. Our study investigates the efficacy of small language models (SLMs) for generating high-quality docstrings by assessing accuracy, conciseness, and clarity, benchmarking performance quantitatively through mathematical formulas and qualitatively through human evaluation using Likert scale. Further, we introduce DocuMint, as a large-scale supervised fine-tuning dataset with 100,000 samples. In quantitative experiments, Llama 3 8B achieved the best performance across all metrics, with conciseness and clarity scores of 0.605 and 64.88, respectively. However, under human evaluation, CodeGemma 7B achieved the highest overall score with an average of 8.3 out of 10 across all metrics. Fine-tuning the CodeGemma 2B model using the DocuMint dataset led to significant improvements in performance across all metrics, with gains of up to 22.5% in conciseness. The fine-tuned model and the dataset can be found in HuggingFace and the code can be found in the repository.

DocuMint: Docstring Generation for Python using Small Language Models

TL;DR

DocuMint addresses the gap in evaluating docstring quality generated by small language models by proposing a dedicated evaluation framework and a large supervised dataset of 100,000 Python function-docstring pairs. The study benchmarks multiple CodeLLMs on three metrics—accuracy, conciseness, and clarity—using both automated scores (notably with BERTScore and readability measures) and human judgments, finding that Llama3 8B performs strongest on quantitative metrics while CodeGemma 7B shines in human evaluations. A LoRA-based fine-tuning of CodeGemma 2B using the DocuMint data yields substantial improvements in conciseness and overall docstring quality, demonstrating the value of dataset-driven fine-tuning. The authors release the DocuMint dataset, the fine-tuned CodeGemma 2B model, and the associated code to support further research and practical adoption in software documentation tooling.

Abstract

Effective communication, specifically through documentation, is the beating heart of collaboration among contributors in software development. Recent advancements in language models (LMs) have enabled the introduction of a new type of actor in that ecosystem: LM-powered assistants capable of code generation, optimization, and maintenance. Our study investigates the efficacy of small language models (SLMs) for generating high-quality docstrings by assessing accuracy, conciseness, and clarity, benchmarking performance quantitatively through mathematical formulas and qualitatively through human evaluation using Likert scale. Further, we introduce DocuMint, as a large-scale supervised fine-tuning dataset with 100,000 samples. In quantitative experiments, Llama 3 8B achieved the best performance across all metrics, with conciseness and clarity scores of 0.605 and 64.88, respectively. However, under human evaluation, CodeGemma 7B achieved the highest overall score with an average of 8.3 out of 10 across all metrics. Fine-tuning the CodeGemma 2B model using the DocuMint dataset led to significant improvements in performance across all metrics, with gains of up to 22.5% in conciseness. The fine-tuned model and the dataset can be found in HuggingFace and the code can be found in the repository.
Paper Structure (23 sections, 3 equations, 5 figures, 4 tables)

This paper contains 23 sections, 3 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Training
  5. Tasks
  6. Methodology
  7. Selected Models
  8. Benchmarking Data
  9. Fine Tuning Data
  10. Fine Tuning
  11. Experiments
  12. Experiment Metrics
  13. Summary statistics of the Human Population
  14. Experiment Setup
  15. Experiment Results
  16. ...and 8 more sections

Figures (5)

  • Figure 1: An example of docstring for a python function. Docstrings serve as documentation and help developers understand how to use the respective code.
  • Figure 2: System prompt that is attached before the function definition when querying the model for inference.
  • Figure 3: Survey participants' frequency in reading docstrings. About 42$\%$ read docstrings several times a week, $\sim$ 17$\%$ read docstrings once a week and several times a month each, while a quarter of them reads docstring once a month.
  • Figure 4: The average weighted scores from the human questionnaire. Instruct variants used for CodeGemma, DeepSeek, and Llama3. CodeGemma performs the best overall surpassing all other models in terms of accuracy and concisenes. Llama3 achieves the best clarity score. StarCoder2 was the second-best-performing model from a human perspective.
  • Figure 5: The loss curve during fine-tuning of the CodeGemma-$2$B model. The loss decreases sharply during the first epoch after which the curve saturates with marginal decrements per epoch.