Evaluating perturbation robustnessof generative systems that use COBOL code inputs

TL;DR

This work tackles robustness of generative systems that take COBOL code as input by introducing a meaning-preserving perturbation framework and a benchmark expansion for COBOL-to-Java translation. It defines a formal setup with inputs $X$, perturbations $\Delta$, outputs $f(X)$, and metrics $\mu$, and uses perturbation-based evaluations to compare $\mu(f(X))$ and $\mu(f(\Delta(X)))$ via proxies like $\bar{\delta}$. The authors build a robustness benchmark by extending the genapp21 corpus to 90 programs and 210 paragraphs through $R_r$ renamings and $R_s$ syntactic variants, enabling single-layer perturbations and comprehensive metric tracking. Key findings show that identifier renaming (especially obfuscation) and certain whitespace/line-break perturbations significantly affect output metrics, with an overall robustness rate of about $0.29$ for any metric change; they also demonstrate dynamic dashboards to diagnose root causes and suggest preprocessing steps to mitigate brittleness. The framework is generalizable to other code tasks (generation, explanation, or modernization) beyond COBOL translation and provides practical guidance for improving reliability in real-world software modernization pipelines.

Abstract

Systems incorporating large language models (LLMs) as a component are known to be sensitive (i.e., non-robust) to minor input variations that do not change the meaning of the input; such sensitivity may reduce the system's usefulness. Here, we present a framework to evaluate robustness of systems using COBOL code as input; our application is translation between COBOL and Java programming languages, but the approach extends to other tasks such as code generation or explanation. Targeting robustness of systems with COBOL as input is essential yet challenging. Many business-critical applications are written in COBOL, yet these are typically proprietary legacy applications and their code is unavailable to LLMs for training. We develop a library of COBOL paragraph and full-program perturbation methods, and create variant-expanded versions of a benchmark dataset of examples for a specific task. The robustness of the LLM-based system is evaluated by measuring changes in values of individual and aggregate metrics calculated on the system's outputs. Finally, we present a series of dynamic table and chart visualization dashboards that assist in debugging the system's outputs, and monitoring and understanding root causes of the system's sensitivity to input variation. These tools can be further used to improve the system by, for instance, indicating variations that should be handled by pre-processing steps.

Figures (5)

• Figure 1: Flowchart of perturbation pipeline for an input code example $X$ and desired perturbation method $\Delta$.
• Figure 2: Top-line statistical summary. Top: Average probability of any input changing a metric $\mu$ (checker). Middle, bottom: Average category- ($D$) and perturbation method-conditional probability of changing any metric ($\bar{\delta}_D$).
• Figure 3: By-method (top) and category (bottom) probabilities of each evaluation metric changing value.
• Figure 4: Input-level statistical summary: Average probability of each/any metric changing, on perturbations of each input $X_i$.
• Figure 6: Partial screenshots of the debug dashboard. Top: Table of boolean metric values; red shading means there was a change relative to the original. Middle, bottom: Partial view of the input COBOL code and Java translations, with highlights showing the changes.