Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging Large Language Models to Improve REST API Testing

Myeongsoo Kim, Tyler Stennett, Dhruv Shah, Saurabh Sinha, Alessandro Orso

TL;DR

The paper tackles the challenge of automated REST API testing when NL descriptions in API specifications contain valuable constraints not captured by standard tooling. It introduces RESTGPT, an LLM-based pipeline that parses OpenAPI specs, generates machine-interpretable rules, and augments the spec with constraints and example values derived from NL descriptions. Empirical results show RESTGPT achieves superior precision (up to 97%) and broader value-generation coverage (73% of parameters) compared with NLP2REST and ARTE, without requiring a validation stage. The work demonstrates a promising direction for enhancing REST API testing via context-aware language models and outlines concrete avenues for further improvement, including fine-tuning, lightweight models, and expanded fault-detection capabilities.

Abstract

The widespread adoption of REST APIs, coupled with their growing complexity and size, has led to the need for automated REST API testing tools. Current tools focus on the structured data in REST API specifications but often neglect valuable insights available in unstructured natural-language descriptions in the specifications, which leads to suboptimal test coverage. Recently, to address this gap, researchers have developed techniques that extract rules from these human-readable descriptions and query knowledge bases to derive meaningful input values. However, these techniques are limited in the types of rules they can extract and prone to produce inaccurate results. This paper presents RESTGPT, an innovative approach that leverages the power and intrinsic context-awareness of Large Language Models (LLMs) to improve REST API testing. RESTGPT takes as input an API specification, extracts machine-interpretable rules, and generates example parameter values from natural-language descriptions in the specification. It then augments the original specification with these rules and values. Our evaluations indicate that RESTGPT outperforms existing techniques in both rule extraction and value generation. Given these promising results, we outline future research directions for advancing REST API testing through LLMs.

Leveraging Large Language Models to Improve REST API Testing

TL;DR

The paper tackles the challenge of automated REST API testing when NL descriptions in API specifications contain valuable constraints not captured by standard tooling. It introduces RESTGPT, an LLM-based pipeline that parses OpenAPI specs, generates machine-interpretable rules, and augments the spec with constraints and example values derived from NL descriptions. Empirical results show RESTGPT achieves superior precision (up to 97%) and broader value-generation coverage (73% of parameters) compared with NLP2REST and ARTE, without requiring a validation stage. The work demonstrates a promising direction for enhancing REST API testing via context-aware language models and outlines concrete avenues for further improvement, including fine-tuning, lightweight models, and expanded fault-detection capabilities.

Abstract

The widespread adoption of REST APIs, coupled with their growing complexity and size, has led to the need for automated REST API testing tools. Current tools focus on the structured data in REST API specifications but often neglect valuable insights available in unstructured natural-language descriptions in the specifications, which leads to suboptimal test coverage. Recently, to address this gap, researchers have developed techniques that extract rules from these human-readable descriptions and query knowledge bases to derive meaningful input values. However, these techniques are limited in the types of rules they can extract and prone to produce inaccurate results. This paper presents RESTGPT, an innovative approach that leverages the power and intrinsic context-awareness of Large Language Models (LLMs) to improve REST API testing. RESTGPT takes as input an API specification, extracts machine-interpretable rules, and generates example parameter values from natural-language descriptions in the specification. It then augments the original specification with these rules and values. Our evaluations indicate that RESTGPT outperforms existing techniques in both rule extraction and value generation. Given these promising results, we outline future research directions for advancing REST API testing through LLMs.
Paper Structure (14 sections, 2 figures, 2 tables)

This paper contains 14 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Motivating Example
  3. REST APIs and OpenAPI Specification
  4. REST API Testing and Assistant Tools
  5. Large Language Model
  6. Motivating Example
  7. Our Approach
  8. Overview
  9. Rule Generator
  10. Specification Enhancement
  11. Preliminary Results
  12. Evaluation Methodology
  13. Results and Discussion
  14. Future Plans

Figures (2)

  • Figure 1: A part of FDIC Bank Data's OpenAPI specification.
  • Figure 2: Overview of our approach.