AccessFixer: Enhancing GUI Accessibility for Low Vision Users With R-GCN Model
Mengxi Zhang, Huaxiao Liu, Chunyang Chen, Guangyong Gao, Han Li, Jian Zhao
TL;DR
This work tackles the persistent problem of GUI accessibility for low-vision users by introducing AccessFixer, a system that converts GUIs into relational graphs and uses a Relational Graph Convolutional Network to jointly fix small component sizes, narrow intervals, and low color contrast. A pre-trained R-GCN encoder–decoder learns from well-annotated GUIs, producing spectral node signals that map to concrete attribute adjustments via polynomial regression, ensuring coordinated changes across related components. Evaluations on 30 real-world apps show an average $81.2\%$ reduction in detected accessibility issues, with multiple cases where all issues were resolved and open-source PRs demonstrating developer adoption. A user study with 10 low-vision participants confirms improved visibility and usability, underscoring practical impact for developers and end users. The authors also open-source their datasets and code, and demonstrate a workflow for integrating fixes into OSS projects through PRs.
Abstract
The Graphical User Interface (GUI) plays a critical role in the interaction between users and mobile applications (apps), aiming at facilitating the operation process. However, due to the variety of functions and non-standardized design, GUIs might have many accessibility issues, like the size of components being too small or their intervals being narrow. These issues would hinder the operation of low vision users, preventing them from obtaining information accurately and conveniently. Although several technologies and methods have been proposed to address these issues, they are typically confined to issue identification, leaving the resolution in the hands of developers. Moreover, it can be challenging to ensure that the color, size, and interval of the fixed GUIs are appropriately compared to the original ones. In this work, we propose a novel approach named AccessFixer, which utilizes the Relational-Graph Convolutional Neural Network (R-GCN) to simultaneously fix three kinds of accessibility issues, including small sizes, narrow intervals, and low color contrast in GUIs. With AccessFixer, the fixed GUIs would have a consistent color palette, uniform intervals, and adequate size changes achieved through coordinated adjustments to the attributes of related components. Our experiments demonstrate the effectiveness and usefulness of AccessFixer in fixing GUI accessibility issues. After fixing 30 real-world apps, our approach solves an average of 81.2% of their accessibility issues. Also, we apply AccessFixer to 10 open-source apps by submitting the fixed results with pull requests (PRs) on GitHub. The results demonstrate that developers approve of our submitted fixed GUIs, with 8 PRs being merged or under fixing. A user study examines that low vision users host a positive attitude toward the GUIs fixed by our method.