Are your apps accessible? A GCN-based accessibility checker for low vision users

TL;DR

This work tackles the challenge of making mobile apps accessible to low-vision users by moving beyond rule-based GUI scanners to a graph-based learning approach. ALVIN builds GUI-graphs from Android layouts, excludes invisible views, and leverages low-vision user annotations to train a Graph Convolutional Network that classifies each GUI component into one of four accessibility issues or accessibility itself. The method outperforms rule-based baselines (average F1 up to 81.2%), and ablation studies show the importance of feature attributes, two convolutional layers, and the fully connected layer. Utility is demonstrated by submitting 20 open-source issues, with 18 being fixed or under fix, indicating strong practical impact. The work also provides an open dataset and code, aiming to facilitate broader research and tooling for accessible UI design.

Abstract

Context: Accessibility issues (e.g., small size and narrow interval) in mobile applications (apps) lead to obstacles for billions of low vision users in interacting with Graphical User Interfaces (GUIs). Although GUI accessibility scanning tools exist, most of them perform rule-based check relying on complex GUI hierarchies. This might make them detect invisible redundant information, cannot handle small deviations, omit similar components, and is hard to extend. Objective: In this paper, we propose a novel approach, named ALVIN (Accessibility Checker for Low Vision), which represents the GUI as a graph and adopts the Graph Convolutional Neural Networks (GCN) to label inaccessible components. Method: ALVIN removes invisible views to prevent detecting redundancy and uses annotations from low vision users to handle small deviations. Also, the GCN model could consider the relations between GUI components, connecting similar components and reducing the possibility of omission. ALVIN only requires users to annotate the relevant dataset when detecting new kinds of issues. Results: Our experiments on 48 apps demonstrate the effectiveness of ALVIN, with precision of 83.5%, recall of 78.9%, and F1-score of 81.2%, outperforming baseline methods. In RQ2, the usefulness is verified through 20 issues submitted to open-source apps. The RQ3 also illustrates the GCN model is better than other models. Conclusion: To summarize, our proposed approach can effectively detect accessibility issues in GUIs for low vision users, thereby guiding developers in fixing them efficiently.

Figures (9)

• Figure 1: The process of Accessibility Scanner to detect the accessibility issues in GUIs
• Figure 2: The workflow of ALVIN.
• Figure 3: The process of constructing GUI-graphs.
• Figure 4: The attributes of a component-node.
• Figure 5: The GCN multi-classification model.