Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Code Quality with Generative AI: Boosting Developer Warning Compliance

Hansen Chang, Christian DeLozier

TL;DR

This work addresses the problem of developers ignoring static-analysis warnings by leveraging large language models to produce plain-language explanations and actionable fixes. It outlines an experimental framework using the NIST Juliet vulnerability suite, two static-analysis tools (cppcheck and clang-check), and multiple LLMs, with careful prompt engineering to elicit concise explanations and remedies. Through case studies on Divide-by-Zero and Stack-Based Buffer Overflow warnings and an automated test-case extraction pipeline yielding 44 validated examples, the study demonstrates the potential of LLM-augmented warnings to improve bug-fix motivation and capability. The findings point toward practical impact, including planned IDE integration and a forthcoming user study with undergraduates to rigorously evaluate whether clearer warnings increase software quality and reduce technical debt.

Abstract

Programmers have long ignored warnings, especially those generated by static analysis tools, due to the potential for false-positives. In some cases, warnings may be indicative of larger issues, but programmers may not understand how a seemingly unimportant warning can grow into a vulnerability. Because these messages tend to be long and confusing, programmers tend to ignore them if they do not cause readily identifiable issues. Large language models can simplify these warnings, explain the gravity of important warnings, and suggest potential fixes to increase developer compliance with fixing warnings.

Enhancing Code Quality with Generative AI: Boosting Developer Warning Compliance

TL;DR

This work addresses the problem of developers ignoring static-analysis warnings by leveraging large language models to produce plain-language explanations and actionable fixes. It outlines an experimental framework using the NIST Juliet vulnerability suite, two static-analysis tools (cppcheck and clang-check), and multiple LLMs, with careful prompt engineering to elicit concise explanations and remedies. Through case studies on Divide-by-Zero and Stack-Based Buffer Overflow warnings and an automated test-case extraction pipeline yielding 44 validated examples, the study demonstrates the potential of LLM-augmented warnings to improve bug-fix motivation and capability. The findings point toward practical impact, including planned IDE integration and a forthcoming user study with undergraduates to rigorously evaluate whether clearer warnings increase software quality and reduce technical debt.

Abstract

Programmers have long ignored warnings, especially those generated by static analysis tools, due to the potential for false-positives. In some cases, warnings may be indicative of larger issues, but programmers may not understand how a seemingly unimportant warning can grow into a vulnerability. Because these messages tend to be long and confusing, programmers tend to ignore them if they do not cause readily identifiable issues. Large language models can simplify these warnings, explain the gravity of important warnings, and suggest potential fixes to increase developer compliance with fixing warnings.
Paper Structure (13 sections, 4 figures, 1 table)

This paper contains 13 sections, 4 figures, 1 table.

Table of Contents

  1. Experimental Setup
  2. Benchmark Suite
  3. Static Analysis Tools
  4. Prompt Engineering
  5. Language Models
  6. Evaluation
  7. Characterization of Static Analysis Warnings
  8. Case Study: Divide-by-Zero
  9. Case Study: Stack Based Buffer Overflow
  10. User Study Design
  11. Automatic Test Case Extraction
  12. Discussion and Future Work
  13. Conclusion

Figures (4)

  • Figure 1: Overview of Using LLMs to Boost Developer Warning Compliance
  • Figure 2: Divide-by-zero test case - In this simple test case, the divide-by-zero always occurs, triggering a warning from the static analysis tool.
  • Figure 3: Buffer Overflow Test Case - If global is set to 5, the smaller data buffer will be used by the strcpy function, leading to a buffer overflow. In other cases, the variable data is uninitialized, also leading to a buffer overflow.
  • Figure 4: ChatGPT Output regarding the buffer overflow test case with both cppcheck warnings as additional input.