Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Agent LLM Committees for Autonomous Software Beta Testing

Sumanth Bharadwaj Hachalli Karanam, Dhiwahar Adhithya Kennady

TL;DR

The paper tackles the high cost and variability of manual beta testing and single-agent LLM approaches by introducing a multi-agent committee framework with diverse personas and vision-enabled UI understanding. It combines a three-round deliberative voting protocol, vision-first browser automation, and a rigorous, instrumented experimental pipeline to automate testing and regression detection on web applications. Through benchmarks on WebShop and OWASP Juice Shop, the approach achieves substantial gains over single-agent baselines in task success, action reliability, and vulnerability detection, while maintaining reproducibility and real-time feasibility for CI/CD. The work's key contributions include the three-round consensus mechanism, persona-driven diversity, a vision-based testing stack, and an open-source framework that enables practical deployment and reproducible research in automated software testing.

Abstract

Manual software beta testing is costly and time-consuming, while single-agent large language model (LLM) approaches suffer from hallucinations and inconsistent behavior. We propose a multi-agent committee framework in which diverse vision-enabled LLMs collaborate through a three-round voting protocol to reach consensus on testing actions. The framework combines model diversity, persona-driven behavioral variation, and visual user interface understanding to systematically explore web applications. Across 84 experimental runs with 9 testing personas and 4 scenarios, multi-agent committees achieve an 89.5 percent overall task success rate. Configurations with 2 to 4 agents reach 91.7 to 100 percent success, compared to 78.0 percent for single-agent baselines, yielding improvements of 13.7 to 22.0 percentage points. At the action level, the system attains a 93.1 percent success rate with a median per-action latency of 0.71 seconds, enabling real-time and continuous integration testing. Vision-enabled agents successfully identify user interface elements, with navigation and reporting achieving 100 percent success and form filling achieving 99.2 percent success. We evaluate the framework on WebShop and OWASP benchmarks, achieving 74.7 percent success on WebShop compared to a 50.1 percent published GPT-3 baseline, and 82.0 percent success on OWASP Juice Shop security testing with coverage of 8 of the 10 OWASP Top 10 vulnerability categories. Across 20 injected regressions, the committee achieves an F1 score of 0.91 for bug detection, compared to 0.78 for single-agent baselines. The open-source implementation enables reproducible research and practical deployment of LLM-based software testing in CI/CD pipelines.

Multi-Agent LLM Committees for Autonomous Software Beta Testing

TL;DR

The paper tackles the high cost and variability of manual beta testing and single-agent LLM approaches by introducing a multi-agent committee framework with diverse personas and vision-enabled UI understanding. It combines a three-round deliberative voting protocol, vision-first browser automation, and a rigorous, instrumented experimental pipeline to automate testing and regression detection on web applications. Through benchmarks on WebShop and OWASP Juice Shop, the approach achieves substantial gains over single-agent baselines in task success, action reliability, and vulnerability detection, while maintaining reproducibility and real-time feasibility for CI/CD. The work's key contributions include the three-round consensus mechanism, persona-driven diversity, a vision-based testing stack, and an open-source framework that enables practical deployment and reproducible research in automated software testing.

Abstract

Manual software beta testing is costly and time-consuming, while single-agent large language model (LLM) approaches suffer from hallucinations and inconsistent behavior. We propose a multi-agent committee framework in which diverse vision-enabled LLMs collaborate through a three-round voting protocol to reach consensus on testing actions. The framework combines model diversity, persona-driven behavioral variation, and visual user interface understanding to systematically explore web applications. Across 84 experimental runs with 9 testing personas and 4 scenarios, multi-agent committees achieve an 89.5 percent overall task success rate. Configurations with 2 to 4 agents reach 91.7 to 100 percent success, compared to 78.0 percent for single-agent baselines, yielding improvements of 13.7 to 22.0 percentage points. At the action level, the system attains a 93.1 percent success rate with a median per-action latency of 0.71 seconds, enabling real-time and continuous integration testing. Vision-enabled agents successfully identify user interface elements, with navigation and reporting achieving 100 percent success and form filling achieving 99.2 percent success. We evaluate the framework on WebShop and OWASP benchmarks, achieving 74.7 percent success on WebShop compared to a 50.1 percent published GPT-3 baseline, and 82.0 percent success on OWASP Juice Shop security testing with coverage of 8 of the 10 OWASP Top 10 vulnerability categories. Across 20 injected regressions, the committee achieves an F1 score of 0.91 for bug detection, compared to 0.78 for single-agent baselines. The open-source implementation enables reproducible research and practical deployment of LLM-based software testing in CI/CD pipelines.
Paper Structure (24 sections, 6 figures, 2 tables, 1 algorithm)

This paper contains 24 sections, 6 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Multi-Agent Committee Architecture
  5. Three-Round Voting Protocol
  6. Vision-Enabled Browser Automation
  7. Persona-Driven Testing
  8. Safety Validators and Storage Layer
  9. Experimental Framework
  10. Experimental Setup
  11. Models and Configuration
  12. Scenarios, Personas, and Experiments
  13. Statistical Analysis
  14. Implementation
  15. Results
  16. ...and 9 more sections

Figures (6)

  • Figure 1: Multi-Agent Committee Testing Framework. Agents analyze screenshots, participate in three-round voting, and execute consensus actions on the browser.
  • Figure 2: Three-round voting protocol showing the flow from independent proposals through discussion to final consensus.
  • Figure 3: Task success rate and committee agreement vs. committee size. Multi-agent committees (2--4 agents) achieve 91.7--100% success vs. 78.0% for single-agent, with 100% agreement on final actions.
  • Figure 4: Persona performance comparison. Accessibility tester achieves highest success (97.6%), while security personas show strong performance (84--91%) with higher latency due to deeper probing.
  • Figure 5: Action distribution and success rates. Atomic actions (navigate, report) achieve 100% while form filling shows 99.2% and clicking shows 83.5%. Scroll actions show lower success due to timing and state synchronization issues.
  • ...and 1 more figures