Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

COBOLAssist: Analyzing and Fixing Compilation Errors for LLM-Powered COBOL Code Generation

Anh T. V. Dau, Shin Hwei Tan, Jinqiu Yang, Nghi D. Q. Bui, Anh Tuan Nguyen

Abstract

Legacy programming languages such as COBOL (Common Business-Oriented Language) remain critical in business computing. However, maintaining legacy COBOL systems is increasingly challenging due to a declining pool of skilled developers and the persistence of COBOL errors that require deep domain expertise to resolve. This paper investigates the challenges of COBOL compilation errors and introduces a framework leveraging large language models (LLMs) to address these issues. We first categorize the common compilation errors in LLM-generated COBOL code into three groups: incomplete code errors, syntax errors, and type-related errors. We further propose COBOLAssist, a technique to enhance code correctness through iterative repairs guided by compilation feedback. Our evaluation using five LLMs including GPT variants and mAInframer, shows a high prevalence of incorrect program structures and function usage in COBOL programs and demonstrates the effectiveness of COBOLAssist, with the compilation success rates increasing from 29.5\% to 64.38\% for GPT-4o-mini and from 41.8\% to 95.89\% for GPT-4o. It also improves pass@1 significantly, for example from 9.1 to 22.6 for GPT-4. Notably, while mAInframer-34B achieves the highest compilation success rate, its functional correctness remains limited. This research not only highlights the limitations in current LLMs for COBOL but also demonstrates a practical path forward for automated debugging in legacy systems.

COBOLAssist: Analyzing and Fixing Compilation Errors for LLM-Powered COBOL Code Generation

Abstract

Legacy programming languages such as COBOL (Common Business-Oriented Language) remain critical in business computing. However, maintaining legacy COBOL systems is increasingly challenging due to a declining pool of skilled developers and the persistence of COBOL errors that require deep domain expertise to resolve. This paper investigates the challenges of COBOL compilation errors and introduces a framework leveraging large language models (LLMs) to address these issues. We first categorize the common compilation errors in LLM-generated COBOL code into three groups: incomplete code errors, syntax errors, and type-related errors. We further propose COBOLAssist, a technique to enhance code correctness through iterative repairs guided by compilation feedback. Our evaluation using five LLMs including GPT variants and mAInframer, shows a high prevalence of incorrect program structures and function usage in COBOL programs and demonstrates the effectiveness of COBOLAssist, with the compilation success rates increasing from 29.5\% to 64.38\% for GPT-4o-mini and from 41.8\% to 95.89\% for GPT-4o. It also improves pass@1 significantly, for example from 9.1 to 22.6 for GPT-4. Notably, while mAInframer-34B achieves the highest compilation success rate, its functional correctness remains limited. This research not only highlights the limitations in current LLMs for COBOL but also demonstrates a practical path forward for automated debugging in legacy systems.

Paper Structure

This paper contains 21 sections, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. COBOL (Common Business Oriented Language)
  4. Code Generation using Large Language Models
  5. Methodology
  6. Construction of COBOL Compilation Error Categorization
  7. COBOL Compilation Error Categorization
  8. COBOLAssist
  9. Evaluation
  10. Experimental Setup
  11. Experiment Results
  12. RQ1: What are the major categories of compilation errors in LLM-generated COBOL code?
  13. RQ2: How effective is COBOLAssist in resolving COBOL compilation errors?
  14. RQ3: What types of compilation errors remain unresolved after using COBOLAssist?
  15. Threats to Validity
  16. ...and 6 more sections

Figures (7)

  • Figure 1: Structure of a COBOL program.
  • Figure 2: A COBOL program with two LINKAGE SECTION declarations, which violates COBOL's structural rules, leading to a syntax error at line 41.
  • Figure 3: An example of Incorrect Use of Built-in Functions. The figure presents a COBOL snippet that incorrectly applies the MOD function as an inline operator rather than using the correct syntax FUNCTION MOD(arg1 arg2).
  • Figure 4: Overview of COBOLAssist—a self-debugging technique with compilation feedback.
  • Figure 5: An input instance from the COBOLEval benchmark.
  • ...and 2 more figures