What the Fix? A Study of ASATs Rule Documentation

TL;DR

This study investigates how ASAT rule documentation is written and how well it supports developers. It develops a three-part nomenclature and a 15-concept taxonomy (What/Why/Fix with Text/Code/Hyperlinks) from over 100 rules across 16 ASATs, then validates and applies it to real-world rules. A survey with 85 respondents evaluating 289 rule-documentations reveals that the What and Fix are generally well-documented while the Why is often missing or of lower quality, highlighting learning and time-saving benefits when Why is included. The work informs practical guidelines for improving ASAT documentation and provides a replication package to extend and refine these findings in future research.

Abstract

Automatic Static Analysis Tools (ASATs) are widely used by software developers to diffuse and enforce coding practices. Yet, we know little about the documentation of ASATs, despite it being critical to learn about the coding practices in the first place. We shed light on this through several contributions. First, we analyze the documentation of more than 100 rules of 16 ASATs for multiple programming languages, and distill a taxonomy of the purposes of the documentation-What triggers a rule; Why it is important; and how to Fix an issue-and its types of contents. Then, we conduct a survey to assess the effectiveness of the documentation in terms of its goals and types of content. We highlight opportunities for improvement in ASAT documentation. In particular, we find that the Why purpose is missing in half of the rules we survey; moreover, when the Why is present, it is more likely to have quality issues than the What and the Fix.

Figures (3)

• Figure 1: Documentation of the rule pointless-statement from Pylint
• Figure 2: Taxonomy applied to rule MultipleVariableDeclarations from Checkstyle
• Figure 3: Participants' evaluation of the usefulness, importance, and quality of the content types and purposes