Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LlamaRestTest: Effective REST API Testing with Small Language Models

Myeongsoo Kim, Saurabh Sinha, Alessandro Orso

TL;DR

LlamaRestTest tackles REST API testing by leveraging two fine-tuned, small LLMs to discover inter-parameter dependencies and generate realistic input values, while incorporating dynamic server feedback through reinforcement learning. By applying PEFT (QLoRA) and multi-bit quantization (2/4/8-bit) to Llama3-8B, the approach balances accuracy and efficiency, enabling runtime adaptation to server responses. Across 12 real-world services, including Spotify, LlamaRestTest achieves higher code and operation coverage and detects more internal server errors than state-of-the-art tools such as RESTler, EvoMaster, MoRest, and ARAT-RL, outperforming RESTGPT in both IPD detection and value generation. The work demonstrates that small, cost-efficient models, guided by server feedback and structured IPD/existing value datasets, can rival or exceed larger models in REST API testing, with practical implications for scalable, open-source testing workflows.

Abstract

Modern web services rely heavily on REST APIs, typically documented using the OpenAPI specification. The widespread adoption of this standard has resulted in the development of many black-box testing tools that generate tests based on OpenAPI specifications. Although Large Language Models (LLMs) have shown promising test-generation abilities, their application to REST API testing remains mostly unexplored. We present LlamaRestTest, a novel approach that employs two custom LLMs-created by fine-tuning and quantizing the Llama3-8B model using mined datasets of REST API example values and inter-parameter dependencies-to generate realistic test inputs and uncover inter-parameter dependencies during the testing process by analyzing server responses. We evaluated LlamaRestTest on 12 real-world services (including popular services such as Spotify), comparing it against RESTGPT, a GPT-powered specification-enhancement tool, as well as several state-of-the-art REST API testing tools, including RESTler, MoRest, EvoMaster, and ARAT-RL. Our results demonstrate that fine-tuning enables smaller models to outperform much larger models in detecting actionable parameter-dependency rules and generating valid inputs for REST API testing. We also evaluated different tool configurations, ranging from the base Llama3-8B model to fine-tuned versions, and explored multiple quantization techniques, including 2-bit, 4-bit, and 8-bit integer formats. Our study shows that small language models can perform as well as, or better than, large language models in REST API testing, balancing effectiveness and efficiency. Furthermore, LlamaRestTest outperforms state-of-the-art REST API testing tools in code coverage achieved and internal server errors identified, even when those tools use RESTGPT-enhanced specifications.

LlamaRestTest: Effective REST API Testing with Small Language Models

TL;DR

LlamaRestTest tackles REST API testing by leveraging two fine-tuned, small LLMs to discover inter-parameter dependencies and generate realistic input values, while incorporating dynamic server feedback through reinforcement learning. By applying PEFT (QLoRA) and multi-bit quantization (2/4/8-bit) to Llama3-8B, the approach balances accuracy and efficiency, enabling runtime adaptation to server responses. Across 12 real-world services, including Spotify, LlamaRestTest achieves higher code and operation coverage and detects more internal server errors than state-of-the-art tools such as RESTler, EvoMaster, MoRest, and ARAT-RL, outperforming RESTGPT in both IPD detection and value generation. The work demonstrates that small, cost-efficient models, guided by server feedback and structured IPD/existing value datasets, can rival or exceed larger models in REST API testing, with practical implications for scalable, open-source testing workflows.

Abstract

Modern web services rely heavily on REST APIs, typically documented using the OpenAPI specification. The widespread adoption of this standard has resulted in the development of many black-box testing tools that generate tests based on OpenAPI specifications. Although Large Language Models (LLMs) have shown promising test-generation abilities, their application to REST API testing remains mostly unexplored. We present LlamaRestTest, a novel approach that employs two custom LLMs-created by fine-tuning and quantizing the Llama3-8B model using mined datasets of REST API example values and inter-parameter dependencies-to generate realistic test inputs and uncover inter-parameter dependencies during the testing process by analyzing server responses. We evaluated LlamaRestTest on 12 real-world services (including popular services such as Spotify), comparing it against RESTGPT, a GPT-powered specification-enhancement tool, as well as several state-of-the-art REST API testing tools, including RESTler, MoRest, EvoMaster, and ARAT-RL. Our results demonstrate that fine-tuning enables smaller models to outperform much larger models in detecting actionable parameter-dependency rules and generating valid inputs for REST API testing. We also evaluated different tool configurations, ranging from the base Llama3-8B model to fine-tuned versions, and explored multiple quantization techniques, including 2-bit, 4-bit, and 8-bit integer formats. Our study shows that small language models can perform as well as, or better than, large language models in REST API testing, balancing effectiveness and efficiency. Furthermore, LlamaRestTest outperforms state-of-the-art REST API testing tools in code coverage achieved and internal server errors identified, even when those tools use RESTGPT-enhanced specifications.
Paper Structure (24 sections, 1 equation, 4 figures, 7 tables)

This paper contains 24 sections, 1 equation, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background and Motivating Example
  3. REST APIs and OpenAPI Specification
  4. REST API Testing and Tools for Enhancing OpenAPI Specifications
  5. Large Language Models
  6. Motivating Example
  7. Our Approach
  8. Base Model Selection
  9. Database Construction
  10. Fine-Tuning and Quantization
  11. Dataset Preparation
  12. Parameter-Efficient Fine-Tuning
  13. Quantization
  14. REST API Testing
  15. Evaluation
  16. ...and 9 more sections

Figures (4)

  • Figure 1: A Part of Ohsome API's OpenAPI Specification.
  • Figure 2: Overview of our approach.
  • Figure 3: Code coverage achieved on 10 runs for open-source services using LlamaRestTest configured with the base Llama3-8B model, the fine-tuned model (FT), and the quantized models (Q).
  • Figure 4: Code coverage achieved on 10 runs for open-source services with MoRest, RESTler, ARAT-RL, EvoMaster, and LlamaRestTest (fine-tuned with 8-bit quantization).