TaskAudit: Detecting Functiona11ity Errors in Mobile Apps via Agentic Task Execution

TL;DR

TaskAudit introduces Task-Au-dit, a system that detects functiona11ity errors in mobile apps by simulating screen reader-driven interactions. It combines a Task Generator (from screenshots), a Task Executor (with a screen reader proxy and multi-agent planning), and an Accessibility Analyzer to identify errors from interaction traces. In evaluations over real-world apps, TaskAudit detected 48 of 78 functiona11ity errors on 54 screens, outperforming static checkers that identified 4–20 errors, and revealed error patterns such as label-functionality mismatches, cluttered navigation, and inappropriate feedback. The work demonstrates the value of simulating actual screen reader user experiences for more comprehensive automated accessibility testing and suggests hybrid approaches combining agentic analysis with traditional crawling for even broader coverage.

Abstract

Accessibility checkers are tools in support of accessible app development and their use is encouraged by accessibility best practices. However, most current checkers evaluate static or mechanically-generated contexts, failing to capture common accessibility errors impacting mobile app functionality. We present TaskAudit, an accessibility evaluation system that focuses on detecting functiona11ity errors through simulated interactions. TaskAudit comprises three components: a Task Generator that constructs interactive tasks from app screens, a Task Executor that uses agents with a screen reader proxy to perform these tasks, and an Accessibility Analyzer that detects and reports accessibility errors by examining interaction traces. Evaluation on real-world apps shows that our strategy detects 48 functiona11ity errors from 54 app screens, compared to between 4 and 20 with existing checkers. Our analysis demonstrates common error patterns that TaskAudit can detect in addition to prior work, including label-functionality mismatch, cluttered navigation, and inappropriate feedback.

Figures (9)

• Figure 1: Exemplar errors from the five functiona11ity error categories that Task-Au-dit detected: Locatability, Actionability, Label, Feedback, and Navigation. "Swipe-R 20" means 20 swipe right gestures to traverse over 20 elements.
• Figure 2: Task-Au-dit's system architecture, comprising a Task Generator, Task Executor, and Accessibility Analyzer. The Task Generator produces task specifications from an initial screenshot. The Task Executor accepts a task and utilizes a multi-agent approach to execute it via a screen reader proxy on an Android emulator. Execution traces and related metadata are sent to the Accessibility Analyzer for evaluation.
• Figure 3: Task-Au-dit's Task Generator parses the interactive elements from a screenshot and generates task specifications. For brevity, we only present selected examples from the specifications, including desc for description, prereq for prerequisites, elem for element, and crit for success criterion.
• Figure 4: Task-Au-dit's Task Executor executes a task which ultimately leads to a functiona11ity error. In each step, the Decision agent generates an action, the transcripts are collected through the screen reader proxy, and the Reflection agent assesses the outcome. Throughout the process, the accessibility focus changes. The last focused element in each step is highlighted in the transcripts. This example includes an input control that does not respond to activation (Step c), which led the Decision agent to attempt other pathways (Step d). However, this proved to be impossible and the execution ultimately terminated after a loop was detected (not shown for brevity).
• Figure 5: Task-Au-dit's Accessibility Analyzer processes one step in the execution trace from Figure \ref{['fig:agent_exec']} [(b) to (c)]. It uses a two-stage prompting process to identify the root cause and reports an error.