DocPrism: Local Categorization and External Filtering to Identify Relevant Code-Documentation Inconsistencies

TL;DR

Code-documentation inconsistencies degrade software quality. DocPrism leverages untuned large language models with a Local Categorization External Filtering approach to detect and explain inconsistencies across Python, TypeScript, C++, and Java without fine-tuning. By reframing inconsistency detection as local completion tasks and using a JSON-driven prompt schema plus post hoc filtering, DocPrism achieves a low flag rate around 15% and a precision around 0.62 across languages, while reducing under-promises through LC-EF. Ablation studies show that Local Categorization and External Filtering outperform naive prompt strategies, and the approach delivers developer-friendly HTML reports to aid manual verification and remediation.

Abstract

Code-documentation inconsistencies are common and undesirable: they can lead to developer misunderstandings and software defects. This paper introduces DocPrism, a multi-language, code-documentation inconsistency detection tool. DocPrism uses a standard large language model (LLM) to analyze and explain inconsistencies. Plain use of LLMs for this task yield unacceptably high false positive rates: LLMs identify natural gaps between high-level documentation and detailed code implementations as inconsistencies. We introduce and apply the Local Categorization, External Filtering (LCEF) methodology to reduce false positives. LCEF relies on the LLM's local completion skills rather than its long-term reasoning skills. In our ablation study, LCEF reduces DocPrism's inconsistency flag rate from 98% to 14%, and increases accuracy from 14% to 94%. On a broad evaluation across Python, TypeScript, C++, and Java, DocPrism maintains a low flag rate of 15%, and achieves a precision of 0.62 without performing any fine-tuning.

Figures (2)

• Figure 1: Three broad kinds of inconsistencies between function-level documentation and its source code implementation.
• Figure 2: Schematic workflow of DocPrism. Section \ref{['sec:approach']} explains each of its components in detail.