Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LELANTE: LEveraging LLM for Automated ANdroid TEsting

Shamit Fatin, Mehbubul Hasan Al-Quvi, Haz Sameen Shahgir, Sukarna Barua, Anindya Iqbal, Sadia Sharmin, Md. Mostofa Akbar, Kallol Kumar Pal, A. Asif Al Rashid

TL;DR

LELANTE addresses the challenge of testing Android apps without manually scripting interactions by using large language models to interpret natural language test case descriptions and autonomously drive GUI actions. It introduces a GUI refinement pipeline to produce a structured, noise-reduced screen representation, and a structured, chain-of-thought prompting strategy that enables iterative action generation and error recovery through backtracking. The framework leverages Appium for on-device execution and employs distillation via LoRA to reduce computational cost, achieving a 73% test execution success rate and 78% error recovery across 390 test cases from 10 apps. The work demonstrates the practical potential of LLM-driven automation for scalable mobile testing, while identifying limitations such as final-state verification and backtracking efficiency, and outlining directions for future improvements. Overall, LELANTE bridges natural-language test descriptions and automated, adaptable test execution, enabling scalable testing workflows with reduced manual scripting effort.

Abstract

Given natural language test case description for an Android application, existing testing approaches require developers to manually write scripts using tools such as Appium and Espresso to execute the corresponding test case. This process is labor-intensive and demands significant effort to maintain as UI interfaces evolve throughout development. In this work, we introduce LELANTE, a novel framework that utilizes large language models (LLMs) to automate test case execution without requiring pre-written scripts. LELANTE interprets natural language test case descriptions, iteratively generate action plans, and perform the actions directly on the Android screen using its GUI. LELANTE employs a screen refinement process to enhance LLM interpretability, constructs a structured prompt for LLMs, and implements an action generation mechanism based on chain-of-thought reasoning of LLMs. To further reduce computational cost and enhance scalability, LELANTE utilizes model distillation using a foundational LLM. In experiments across 390 test cases spanning 10 popular Android applications, LELANTE achieved a 73% test execution success rate. Our results demonstrate that LLMs can effectively bridge the gap between natural language test case description and automated execution, making mobile testing more scalable and adaptable.

LELANTE: LEveraging LLM for Automated ANdroid TEsting

TL;DR

LELANTE addresses the challenge of testing Android apps without manually scripting interactions by using large language models to interpret natural language test case descriptions and autonomously drive GUI actions. It introduces a GUI refinement pipeline to produce a structured, noise-reduced screen representation, and a structured, chain-of-thought prompting strategy that enables iterative action generation and error recovery through backtracking. The framework leverages Appium for on-device execution and employs distillation via LoRA to reduce computational cost, achieving a 73% test execution success rate and 78% error recovery across 390 test cases from 10 apps. The work demonstrates the practical potential of LLM-driven automation for scalable mobile testing, while identifying limitations such as final-state verification and backtracking efficiency, and outlining directions for future improvements. Overall, LELANTE bridges natural-language test descriptions and automated, adaptable test execution, enabling scalable testing workflows with reduced manual scripting effort.

Abstract

Given natural language test case description for an Android application, existing testing approaches require developers to manually write scripts using tools such as Appium and Espresso to execute the corresponding test case. This process is labor-intensive and demands significant effort to maintain as UI interfaces evolve throughout development. In this work, we introduce LELANTE, a novel framework that utilizes large language models (LLMs) to automate test case execution without requiring pre-written scripts. LELANTE interprets natural language test case descriptions, iteratively generate action plans, and perform the actions directly on the Android screen using its GUI. LELANTE employs a screen refinement process to enhance LLM interpretability, constructs a structured prompt for LLMs, and implements an action generation mechanism based on chain-of-thought reasoning of LLMs. To further reduce computational cost and enhance scalability, LELANTE utilizes model distillation using a foundational LLM. In experiments across 390 test cases spanning 10 popular Android applications, LELANTE achieved a 73% test execution success rate. Our results demonstrate that LLMs can effectively bridge the gap between natural language test case description and automated execution, making mobile testing more scalable and adaptable.
Paper Structure (18 sections, 2 equations, 3 figures, 2 tables)

This paper contains 18 sections, 2 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Problem Statement
  3. Methodology
  4. Screen Representation Refinement
  5. LLM-Driven Action Generation
  6. Action Execution Using GUI Interaction
  7. Distillation for Cost-Effective Execution
  8. Evaluation
  9. Evaluation metrics.
  10. Performance of LELANTE in test case execution.
  11. Impact of Test Case Length on Execution Success
  12. Impact of Ambiguity Resolution in Test Cases
  13. Related Works
  14. Discussion
  15. Conclusion and Future Work
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Overview of the LELANTE test case execution process. Given a natural language test case description, LELANTE incrementally refines the screen GUI representation and executes the test case step by step until completion.
  • Figure 2: Screen representation refinement for LLM understanding. This process extracts essential and actionable elements (e.g., buttons, text fields) from the original screen representation (XML) and convert it to an optimized format suitable for LLM prompts.
  • Figure 3: The left histogram shows test case length distribution, with most cases in the 3–10 step range. The right histogram shows that shorter test cases succeed more often, while longer ones have lower success due to increased interactions and error risks