Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Large Language Models for ABAP Code Generation: An Empirical Study on Iterative Improvement by Compiler Feedback

Stephan Wallraven, Tim Köhne, Hartmut Westenberger, Andreas Moser

TL;DR

The paper presents a benchmark for ABAP code generation by LLMs using 180 tasks that combine HumanEval-like problems with SAP-specific scenarios, automated in a standardized SAP environment. It compares diverse open- and closed-source LLMs under iterative compiler feedback, finding that the most capable models (e.g., GPT-5, Claude-Sonnet-4) can reach about 77% functional success after five rounds, while smaller or less architecture-tuned models show limited gains. Task difficulty is uneven, with List/Array operations posing the biggest challenge, though ABAP Database operations are solvable by at least one model in all cases. The results emphasize the potential of AI-assisted ABAP development through iterative error correction, while also highlighting the need for broader ABAP-focused benchmarks and closer SAP integration to move toward more autonomous generation.

Abstract

This work investigates the performance of Large Language Models (LLMs) in generating ABAP code. Despite successful applications of generative AI in many programming languages, there are hardly any systematic analyses of ABAP code generation to date. The aim of the study is to empirically analyze to what extent various LLMs can generate syntactically correct and functional ABAP code, how effectively they use compiler feedback for iterative improvement, and which task types pose special challenges. For this purpose, a benchmark with 180 tasks is conducted, consisting of adapted HumanEval tasks and practical SAP scenarios. The results show significant performance differences between the models: more powerful LLMs achieve success rates of around 75% after several iterations and benefit greatly from compiler feedback, while smaller models perform significantly weaker. Overall, the study highlights the high potential of powerful LLMs for ABAP development processes, especially in iterative error correction.

Benchmarking Large Language Models for ABAP Code Generation: An Empirical Study on Iterative Improvement by Compiler Feedback

TL;DR

The paper presents a benchmark for ABAP code generation by LLMs using 180 tasks that combine HumanEval-like problems with SAP-specific scenarios, automated in a standardized SAP environment. It compares diverse open- and closed-source LLMs under iterative compiler feedback, finding that the most capable models (e.g., GPT-5, Claude-Sonnet-4) can reach about 77% functional success after five rounds, while smaller or less architecture-tuned models show limited gains. Task difficulty is uneven, with List/Array operations posing the biggest challenge, though ABAP Database operations are solvable by at least one model in all cases. The results emphasize the potential of AI-assisted ABAP development through iterative error correction, while also highlighting the need for broader ABAP-focused benchmarks and closer SAP integration to move toward more autonomous generation.

Abstract

This work investigates the performance of Large Language Models (LLMs) in generating ABAP code. Despite successful applications of generative AI in many programming languages, there are hardly any systematic analyses of ABAP code generation to date. The aim of the study is to empirically analyze to what extent various LLMs can generate syntactically correct and functional ABAP code, how effectively they use compiler feedback for iterative improvement, and which task types pose special challenges. For this purpose, a benchmark with 180 tasks is conducted, consisting of adapted HumanEval tasks and practical SAP scenarios. The results show significant performance differences between the models: more powerful LLMs achieve success rates of around 75% after several iterations and benefit greatly from compiler feedback, while smaller models perform significantly weaker. Overall, the study highlights the high potential of powerful LLMs for ABAP development processes, especially in iterative error correction.
Paper Structure (21 sections, 10 figures, 5 tables)

This paper contains 21 sections, 10 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Problem Statement
  4. Objectives and Research Questions
  5. Related Work
  6. Methodology
  7. Structure and Methodology of the Benchmark
  8. Selection of LLMs
  9. Evaluation Environment
  10. Evaluation Criteria
  11. Investigation
  12. Results
  13. Differences between Feedback Rounds
  14. Differences between Task Types
  15. Error Distribution
  16. ...and 6 more sections

Figures (10)

  • Figure 1: Rough Process of the Benchmark.
  • Figure 2: Benchmark results broken down by model and feedback rounds in percent.
  • Figure 3: Kaplan-Meier survival curves showing the probability of remaining test errors over 0-5 feedback iterations (Basis: 10 x 180 test runs)
  • Figure 4: Benchmark results broken down by model and task focus in percent.
  • Figure 5: Kaplan-Meier survival curves for ABAP Database Operation tasks differentiated by LLM models.
  • ...and 5 more figures