Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automated Testing of the GUI of a Real-Life Engineering Software using Large Language Models

Tim Rosenbach, David Heidrich, Alexander Weinert

TL;DR

End-user acceptance of GUI is hard to capture with automated tests, motivating exploratory testing with LLMs. The authors propose GERALLT, a two-agent system (controller and evaluator) that navigates a Windows desktop application (RCE) via a PyWinAuto-based GUI parser and task-driven prompts. They describe the architecture and report qualitative evaluation with developers, showing GERALLT surfaces unintuitive and inconsistent UI behaviors. The work demonstrates that LLM-driven exploratory testing can reveal blind spots in manual testing and has potential to accelerate left-shifting acceptance testing in engineering software.

Abstract

One important step in software development is testing the finished product with actual users. These tests aim, among other goals, at determining unintuitive behavior of the software as it is presented to the end-user. Moreover, they aim to determine inconsistencies in the user-facing interface. They provide valuable feedback for the development of the software, but are time-intensive to conduct. In this work, we present GERALLT, a system that uses Large Language Models (LLMs) to perform exploratory tests of the Graphical User Interface (GUI) of a real-life engineering software. GERALLT automatically generates a list of potential unintuitive and inconsistent parts of the interface. We present the architecture of GERALLT and evaluate it on a real-world use case of the engineering software, which has been extensively tested by developers and users. Our results show that GERALLT is able to determine issues with the interface that support the software development team in future development of the software.

Automated Testing of the GUI of a Real-Life Engineering Software using Large Language Models

TL;DR

End-user acceptance of GUI is hard to capture with automated tests, motivating exploratory testing with LLMs. The authors propose GERALLT, a two-agent system (controller and evaluator) that navigates a Windows desktop application (RCE) via a PyWinAuto-based GUI parser and task-driven prompts. They describe the architecture and report qualitative evaluation with developers, showing GERALLT surfaces unintuitive and inconsistent UI behaviors. The work demonstrates that LLM-driven exploratory testing can reveal blind spots in manual testing and has potential to accelerate left-shifting acceptance testing in engineering software.

Abstract

One important step in software development is testing the finished product with actual users. These tests aim, among other goals, at determining unintuitive behavior of the software as it is presented to the end-user. Moreover, they aim to determine inconsistencies in the user-facing interface. They provide valuable feedback for the development of the software, but are time-intensive to conduct. In this work, we present GERALLT, a system that uses Large Language Models (LLMs) to perform exploratory tests of the Graphical User Interface (GUI) of a real-life engineering software. GERALLT automatically generates a list of potential unintuitive and inconsistent parts of the interface. We present the architecture of GERALLT and evaluate it on a real-world use case of the engineering software, which has been extensively tested by developers and users. Our results show that GERALLT is able to determine issues with the interface that support the software development team in future development of the software.

Paper Structure

This paper contains 12 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. The Use Case
  4. Testing RCE with LLMs
  5. System Architecture
  6. Controller Prompt
  7. Evaluator Prompt
  8. Evaluation and Known Limitations
  9. Quantitative Evaluation Results
  10. Example Issue
  11. Known Limitations
  12. Conclusion and Future Work

Figures (5)

  • Figure 1: Pages of the tool integration wizard of RCE
  • Figure 2: Architecture of GERALLT. The component "Previous Screenshot" holds the screenshot of the taken during the last iteration. It is replaced by an updated screenshot after each iteration. The dashed line in the bottom right denotes that the evaluator only receives the last action performed on the instead of the complete log.
  • Figure 3: The appearance of the online help in the tool integration of RCE.
  • Figure 4: The evaluation process for our system.
  • Figure 5: Example error, where the evaluator criticized that the error message is too imprecise.