Autonomous QA Agent: A Retrieval-Augmented Framework for Reliable Selenium Script Generation

TL;DR

This work targets the semantic gap between natural language QA requirements and executable Selenium scripts by grounding generation in a Retrieval-Augmented Generation (RAG) framework. It ingests diverse materials (Markdown, PDF, HTML) into a vector store and retrieves both textual requirements and HTML DOM context to condition script generation, dramatically reducing UI element selector hallucinations. On 20 e-commerce test scenarios, the approach achieves $100 ext{%}$ syntax validity and $90 ext{%}$ execution success, outperforming a standard LLM baseline (syntax $95 ext{%}$, execution $30 ext{%}$) with statistically significant gains ($p<0.001$). The study demonstrates that context grounding in DOM structure and documentation can substantially improve automated UI testing, offering a scalable path toward self-writing, self-healing test suites while acknowledging limitations related to dynamic pages and domain generalization.

Abstract

Software testing is critical in the software development lifecycle, yet translating requirements into executable test scripts remains manual and error-prone. While Large Language Models (LLMs) can generate code, they often hallucinate non-existent UI elements. We present the Autonomous QA Agent, a Retrieval-Augmented Generation (RAG) system that grounds Selenium script generation in project-specific documentation and HTML structure. By ingesting diverse formats (Markdown, PDF, HTML) into a vector database, our system retrieves relevant context before generation. Evaluation on 20 e-commerce test scenarios shows our RAG approach achieves 100% (20/20) syntax validity and 90% (18/20, 95% CI: [85%, 95%], p < 0.001) execution success, compared to 30% for standard LLM generation. While our evaluation is limited to a single domain, our method significantly reduces hallucinations by grounding generation in actual DOM structure, demonstrating RAG's potential for automated UI testing.

Figures (13)

• Figure 1: Knowledge Graph of Application Structure. This network graph visualizes the relationships between different pages (blue nodes) and their constituent DOM elements (orange nodes), illustrating the structural complexity handled by the agent.
• Figure 2: t-SNE Projection of DOM Element Embeddings (Perplexity=30, n_iter=1000, random_state=42). This scatter plot visualizes the semantic clustering of different UI elements in the vector space, showing distinct clusters for Buttons, Input Fields, and Text Content.
• Figure 3: Hyperparameter Landscape (Grid: 20×20, Metric: Element Resolution %). A 3D surface plot showing the impact of Retrieval Count ($k$) and Chunk Size on the Element Resolution Accuracy. The peak indicates the optimal configuration region at k=3, chunk_size=1000.
• Figure 4: Sample Prompt Template for RAG-based Script Generation. This figure shows the actual prompt structure used, including system instructions, retrieved context (documentation + HTML), and the user query.
• Figure 5: Example HTML Chunk Retrieved from Vector Store. This shows an actual HTML snippet ingested from the checkout page, containing the structural information (IDs, classes) needed for accurate selector generation.