Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

APITestGenie: Generating Web API Tests from Requirements and API Specifications with LLMs

André Pereira, Bruno Lima, João Pascoal Faria

Abstract

Modern software systems rely heavily on Web APIs, yet creating meaningful and executable test scripts remains a largely manual, time-consuming, and error-prone task. In this paper, we present APITestGenie, a novel tool that leverages Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), and prompt engineering to automatically generate API integration tests directly from business requirements and OpenAPI specifications. We evaluated APITestGenie on 10 real-world APIs, including 8 APIs comprising circa 1,000 live endpoints from an industrial partner in the automotive domain. The tool was able to generate syntactically and semantically valid test scripts for 89\% of the business requirements under test after at most three attempts. Notably, some generated tests revealed previously unknown defects in the APIs, including integration issues between endpoints. Statistical analysis identified API complexity and level of detail in business requirements as primary factors influencing success rates, with the level of detail in API documentation also affecting outcomes. Feedback from industry practitioners confirmed strong interest in adoption, substantially reducing the manual effort in writing acceptance tests, and improving the alignment between tests and business requirements.

APITestGenie: Generating Web API Tests from Requirements and API Specifications with LLMs

Abstract

Modern software systems rely heavily on Web APIs, yet creating meaningful and executable test scripts remains a largely manual, time-consuming, and error-prone task. In this paper, we present APITestGenie, a novel tool that leverages Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), and prompt engineering to automatically generate API integration tests directly from business requirements and OpenAPI specifications. We evaluated APITestGenie on 10 real-world APIs, including 8 APIs comprising circa 1,000 live endpoints from an industrial partner in the automotive domain. The tool was able to generate syntactically and semantically valid test scripts for 89\% of the business requirements under test after at most three attempts. Notably, some generated tests revealed previously unknown defects in the APIs, including integration issues between endpoints. Statistical analysis identified API complexity and level of detail in business requirements as primary factors influencing success rates, with the level of detail in API documentation also affecting outcomes. Feedback from industry practitioners confirmed strong interest in adoption, substantially reducing the manual effort in writing acceptance tests, and improving the alignment between tests and business requirements.

Paper Structure

This paper contains 22 sections, 8 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Solution Design
  4. Architecture and Workflow
  5. System and User Prompt
  6. API Pre-processing and Retrieval-Augmented Generation
  7. Evaluation
  8. Experimental Evaluation
  9. Materials and methods
  10. Overall performance results
  11. Assessment of the generated test scripts
  12. Analysis of influencing factors
  13. Comparison with EvoMaster
  14. User Feedback
  15. Answers to Research Questions
  16. ...and 7 more sections

Figures (8)

  • Figure 1: APITestGenie flow diagram, showcasing the main processes in the system, inputs and outputs.
  • Figure 2: "Context" section of the system prompt.
  • Figure 3: "Performance" section of the system prompt.
  • Figure 4: "Output" section of the system prompt.
  • Figure 5: APITestGenie’s RAG process builds a database from the API specification, expands requirements, and retrieves relevant API segments to give the LLM precise context for generating targeted test scripts.
  • ...and 3 more figures