Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models for Automated Web-Form-Test Generation: An Empirical Study

Tao Li, Chenhui Cui, Rubing Huang, Dave Towey, Lei Ma

TL;DR

This study empirically evaluates 11 LLMs for automated web-form-test generation using 146 forms from 30 open-source Java web apps. It introduces three context-construction methods (Raw HTML, LLM-Processed HTML, Parser-Processed HTML) and three prompt designs, with PH-P yielding the highest average SSR and GPT-4 typically performing best. The results show substantial variation across models and prompts, underscoring the importance of structured context and model choice, while also highlighting practical implications for prompt design, model selection, and tester practices. The work provides actionable guidance for practitioners and lays groundwork for future enhancements, including automated prompt optimization and expansion to more dynamic or sensitive web forms.

Abstract

Testing web forms is an essential activity for ensuring the quality of web applications. It typically involves evaluating the interactions between users and forms. Automated test-case generation remains a challenge for web-form testing: Due to the complex, multi-level structure of web pages, it can be difficult to automatically capture their inherent contextual information for inclusion in the tests. Large Language Models (LLMs) have shown great potential for contextual text generation. This motivated us to explore how they could generate automated tests for web forms, making use of the contextual information within form elements. To the best of our knowledge, no comparative study examining different LLMs has yet been reported for web-form-test generation. To address this gap in the literature, we conducted a comprehensive empirical study investigating the effectiveness of 11 LLMs on 146 web forms from 30 open-source Java web applications. In addition, we propose three HTML-structure-pruning methods to extract key contextual information. The experimental results show that different LLMs can achieve different testing effectiveness. Compared with GPT-4, the other LLMs had difficulty generating appropriate tests for the web forms: Their successfully-submitted rates (SSRs) decreased by 9.10% to 74.15%. Our findings also show that, for all LLMs, when the designed prompts include complete and clear contextual information about the web forms, more effective web-form tests were generated. Specifically, when using Parser-Processed HTML for Task Prompt (PH-P), the SSR averaged 70.63%, higher than the 60.21% for Raw HTML for Task Prompt (RH-P) and 50.27% for LLM-Processed HTML for Task Prompt (LH-P). Finally, this paper also highlights strategies for selecting LLMs based on performance metrics, and for optimizing the prompt design to improve the quality of the web-form tests.

Large Language Models for Automated Web-Form-Test Generation: An Empirical Study

TL;DR

This study empirically evaluates 11 LLMs for automated web-form-test generation using 146 forms from 30 open-source Java web apps. It introduces three context-construction methods (Raw HTML, LLM-Processed HTML, Parser-Processed HTML) and three prompt designs, with PH-P yielding the highest average SSR and GPT-4 typically performing best. The results show substantial variation across models and prompts, underscoring the importance of structured context and model choice, while also highlighting practical implications for prompt design, model selection, and tester practices. The work provides actionable guidance for practitioners and lays groundwork for future enhancements, including automated prompt optimization and expansion to more dynamic or sensitive web forms.

Abstract

Testing web forms is an essential activity for ensuring the quality of web applications. It typically involves evaluating the interactions between users and forms. Automated test-case generation remains a challenge for web-form testing: Due to the complex, multi-level structure of web pages, it can be difficult to automatically capture their inherent contextual information for inclusion in the tests. Large Language Models (LLMs) have shown great potential for contextual text generation. This motivated us to explore how they could generate automated tests for web forms, making use of the contextual information within form elements. To the best of our knowledge, no comparative study examining different LLMs has yet been reported for web-form-test generation. To address this gap in the literature, we conducted a comprehensive empirical study investigating the effectiveness of 11 LLMs on 146 web forms from 30 open-source Java web applications. In addition, we propose three HTML-structure-pruning methods to extract key contextual information. The experimental results show that different LLMs can achieve different testing effectiveness. Compared with GPT-4, the other LLMs had difficulty generating appropriate tests for the web forms: Their successfully-submitted rates (SSRs) decreased by 9.10% to 74.15%. Our findings also show that, for all LLMs, when the designed prompts include complete and clear contextual information about the web forms, more effective web-form tests were generated. Specifically, when using Parser-Processed HTML for Task Prompt (PH-P), the SSR averaged 70.63%, higher than the 60.21% for Raw HTML for Task Prompt (RH-P) and 50.27% for LLM-Processed HTML for Task Prompt (LH-P). Finally, this paper also highlights strategies for selecting LLMs based on performance metrics, and for optimizing the prompt design to improve the quality of the web-form tests.
Paper Structure (45 sections, 1 equation, 9 figures, 8 tables, 2 algorithms)

This paper contains 45 sections, 1 equation, 9 figures, 8 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. HTML Parsing Process
  4. Web-Form testing
  5. Basic Web-Form Testing Framework
  6. Web-Element Location Identification Strategies
  7. LLMs
  8. Approach
  9. HTML Pruning
  10. Context Construction
  11. Context from Raw HTML
  12. Context from LLM-Processed HTML
  13. Context from Parser-Processed HTML
  14. Prompt Design
  15. Global Prompt
  16. ...and 30 more sections

Figures (9)

  • Figure 1: HTML parsing process.
  • Figure 2: Process of using Selenium to control automated testing (black dashed lines represent standard steps; red solid lines are steps requiring WebDriver).
  • Figure 3: Framework of this empirical study.
  • Figure 4: Basic framework of the prompt structure.
  • Figure 5: Example illustrating the entire web-form testing process.
  • ...and 4 more figures