From Struggle to Success: Context-Aware Guidance for Screen Reader Users in Computer Use

TL;DR

AskEase is a context-aware, on-demand assistant that helps screen reader users navigate computer interfaces by integrating environment, knowledge, and conversational contexts to deliver step-by-step, non-visual guidance. Built as an NVDA add-on and powered by an advanced LLM, it uses retrieval-augmented generation and structured prompts to provide precise, user-aligned support with minimal disruption to workflow. In a within-subject study with 12 SR users, AskEase improved task completion and reduced perceived workload compared with baseline tools, demonstrating the practical potential of context-aware AI for accessible computing. The work highlights design implications for personalization, seamless assistance, and trustworthy AI in accessibility tools, and suggests future directions for broader deployment and privacy-preserving deployments.

Abstract

Equal access to digital technologies is critical for education, employment, and social participation. However, mainstream interfaces are visually oriented, creating steep learning curves and frequent obstacles for screen reader users, and limiting their independence and opportunities. Existing support is inadequate -- tutorials mainly target sighted users, while human assistance lacks real-time availability. We introduce AskEase, an on-demand AI assistant that provides step-by-step, screen reader user-friendly guidance for computer use. AskEase manages multiple sources of context to infer user intent and deliver precise, situation-specific guidance. Its seamless interaction design minimizes disruption and reduces the effort of seeking help. We demonstrated its effectiveness through representative usage scenarios and robustness tests. In a within-subjects study with 12 screen reader users, AskEase significantly improved task success while reducing perceived workload, including physical demand, effort, and frustration. These results demonstrate the potential of LLM-powered assistants to promote accessible computing and expand opportunities for users with visual impairments.

Figures (5)

• Figure 1: AskEase dialog: pop-up with a question input field and chat history view, showing one user question and its AI reply at a time, with Previous/Next buttons to navigate through the multi‑turn chat history..
• Figure 2: This scenario illustrates how AskEase supports users in learning and interacting with GitHub Copilot, an emerging AI tool. The interaction unfolds across three turns: onboarding, progress clarification, and actionable follow-up.
• Figure 3: Multi-turn assistance in Word: (1) initial step-by-step guidance for inserting left-aligned page numbers, (2) adaptive guidance to address an accessibility issue where the left-aligned page number style is incorrectly displayed as "Plain number 1", and (3) confirmation of correct page number placement.
• Figure 4: Comparison of task completion and NASA-TLX ratings between baseline tools and AskEase. NASA-TLX scores range from 1 (strong disagreement) to 7 (strong agreement). Higher scores indicate higher perceived workload, except for performance, where higher values reflect better perceived task performance.
• Figure 5: Participants' responses to the post-task 7-point Likert scale survey. Higher scores indicate stronger agreement.