Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

KAT: Dependency-aware Automated API Testing with Large Language Models

Tri Le, Thien Tran, Duy Cao, Vy Le, Tien Nguyen, Vu Nguyen

TL;DR

KAT tackles the challenge of testing RESTful APIs with complex inter-endpoint and inter-parameter dependencies by using GPT-driven prompting to infer operation and parameter relationships from OpenAPI specifications. The approach builds an Operation Dependency Graph, generates operation sequences, and produces test scripts and data that respect these dependencies, enabling automated black-box API testing. Empirical evaluation on 12 real-world services shows that KAT improves overall status-code coverage and better detects undocumented status codes while reducing false positives compared to the state-of-the-art RestTestGen. This demonstrates the practical potential of LLM-guided test generation to enhance API robustness and reduce manual test engineering effort.

Abstract

API testing has increasing demands for software companies. Prior API testing tools were aware of certain types of dependencies that needed to be concise between operations and parameters. However, their approaches, which are mostly done manually or using heuristic-based algorithms, have limitations due to the complexity of these dependencies. In this paper, we present KAT (Katalon API Testing), a novel AI-driven approach that leverages the large language model GPT in conjunction with advanced prompting techniques to autonomously generate test cases to validate RESTful APIs. Our comprehensive strategy encompasses various processes to construct an operation dependency graph from an OpenAPI specification and to generate test scripts, constraint validation scripts, test cases, and test data. Our evaluation of KAT using 12 real-world RESTful services shows that it can improve test coverage, detect more undocumented status codes, and reduce false positives in these services in comparison with a state-of-the-art automated test generation tool. These results indicate the effectiveness of using the large language model for generating test scripts and data for API testing.

KAT: Dependency-aware Automated API Testing with Large Language Models

TL;DR

KAT tackles the challenge of testing RESTful APIs with complex inter-endpoint and inter-parameter dependencies by using GPT-driven prompting to infer operation and parameter relationships from OpenAPI specifications. The approach builds an Operation Dependency Graph, generates operation sequences, and produces test scripts and data that respect these dependencies, enabling automated black-box API testing. Empirical evaluation on 12 real-world services shows that KAT improves overall status-code coverage and better detects undocumented status codes while reducing false positives compared to the state-of-the-art RestTestGen. This demonstrates the practical potential of LLM-guided test generation to enhance API robustness and reduce manual test engineering effort.

Abstract

API testing has increasing demands for software companies. Prior API testing tools were aware of certain types of dependencies that needed to be concise between operations and parameters. However, their approaches, which are mostly done manually or using heuristic-based algorithms, have limitations due to the complexity of these dependencies. In this paper, we present KAT (Katalon API Testing), a novel AI-driven approach that leverages the large language model GPT in conjunction with advanced prompting techniques to autonomously generate test cases to validate RESTful APIs. Our comprehensive strategy encompasses various processes to construct an operation dependency graph from an OpenAPI specification and to generate test scripts, constraint validation scripts, test cases, and test data. Our evaluation of KAT using 12 real-world RESTful services shows that it can improve test coverage, detect more undocumented status codes, and reduce false positives in these services in comparison with a state-of-the-art automated test generation tool. These results indicate the effectiveness of using the large language model for generating test scripts and data for API testing.
Paper Structure (27 sections, 1 equation, 11 figures, 4 tables, 1 algorithm)

This paper contains 27 sections, 1 equation, 11 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivating Example
  3. Observations
  4. The state-of-the-art Approaches
  5. Capability of GPT and Motivation
  6. Dependency-aware Automated API Testing
  7. Approach Overview
  8. Operation Dependency Graph Construction (Fig. \ref{['fig:ODG construction']})
  9. Important Concepts
  10. ODG Construction Algorithm
  11. Test Data Generation
  12. Generating Test Scripts and Test Cases
  13. Empirical Evaluation
  14. Research Questions
  15. Dataset
  16. ...and 12 more sections

Figures (11)

  • Figure 1: An Example of OpenAPI/Swagger Specification (OAS) file
  • Figure 2: KAT: Dependency-aware Automated API Testing
  • Figure 3: ODG Construction
  • Figure 4: Dependence between GET/flights and POST/booking in ODG
  • Figure 5: Example of an Operation-Schema dependency dictionary. The operation POST /booking has two Operation-Schema dependencies with the schemas "Flight" and "Booking". This is indicated by the pairs of parameter flightId (line 22) with field id (line 55) and flight (line 64) in Fig \ref{['fig:example']}.
  • ...and 6 more figures

Theorems & Definitions (3)

  • Definition 1: Operation Dependency Graph
  • Definition 2: Operation-Schema Dependency
  • Definition 3: Schema-Schema Dependency