LogiAgent: Automated Logical Testing for REST Systems with LLM-Based Multi-Agents

TL;DR

LogiAgent tackles the gap in REST API testing by focusing on logical correctness rather than only server crashes. It introduces an LLM-driven multi-agent system with a Test Scenario Generator, API Request Executor, and API Response Validator, coordinated by a Scenario Scheduler and reinforced by long-term Execution Memory. Across 12 real-world REST systems, LogiAgent detects $234$ true logical issues with an accuracy of $66.19\%$, identifies $49$ server crashes, and achieves superior test coverage compared with four baselines; ablation confirms the memory component boosts coverage. The work demonstrates that integrating business-logic-aware oracles and memory-driven adaptation enables more thorough validation of REST APIs and offers practical gains for real-world software quality assurance.

Abstract

Automated testing for REST APIs has become essential for ensuring the correctness and reliability of modern web services. While existing approaches primarily focus on detecting server crashes and error codes, they often overlook logical issues that arise due to evolving business logic and domain-specific requirements. To address this limitation, we propose LogiAgent, a novel approach for logical testing of REST systems. Built upon a large language model (LLM)-driven multi-agent framework, LogiAgent integrates a Test Scenario Generator, API Request Executor, and API Response Validator to collaboratively generate, execute, and validate API test scenarios. Unlike traditional testing methods that focus on status codes like 5xx, LogiAgent incorporates logical oracles that assess responses based on business logic, ensuring more comprehensive testing. The system is further enhanced by an Execution Memory component that stores historical API execution data for contextual consistency. We conduct extensive experiments across 12 real-world REST systems, demonstrating that LogiAgent effectively identifies 234 logical issues with an accuracy of 66.19%. Additionally, it basically excels in detecting server crashes and achieves superior test coverage compared to four state-of-the-art REST API testing tools. An ablation study confirms the significant contribution of LogiAgent's memory components to improving test coverage.

Figures (7)

• Figure 1: The OpenAPI Specification and Example Response of /pet API
• Figure 2: Approach Overview of LogiAgent
• Figure 3: Prompt Template Used in Test Scenario Generator
• Figure 4: Prompt Template Used in API Request Executor
• Figure 5: Prompt Template Used in API Response Validator