Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM impact on BLV programming

Prashant Chandrasekar, Mariel Couvillion, Ayshwarya Saktheeswaran, Jessica Zeitz

TL;DR

This work investigates how Large Language Models (LLMs) and GenAI affect blind and low-vision (BLV) developers in programming. Through a literature review and a dual evaluation—an empirical assessment of five LLM-powered IDEs across core coding tasks and a user study with five BLV programmers—the authors identify both potential benefits and new accessibility barriers. They report inconsistent outputs across IDEs, unresolved workflow fragmentation, and AI hallucinations that complicate debugging and verification for BLV users. The findings highlight a need for accessibility-centered design, standardized AI outputs, and closer collaboration with BLV users to ensure that generative AI tools meaningfully improve programming experiences for the BLV community.

Abstract

Large Language Models (LLMs) are rapidly becoming integral to a wide range of tools, tasks, and problem-solving processes, especially in software development. Originally designed for natural language processing tasks such as text generation, LLMs are increasingly being used to assist both professionals and students in writing code. This growing reliance on LLM-based tools is reshaping programming workflows and task execution. In this study, we explore the impact of these technologies on blind and low-vision (BLV) developers. Our review of existing literature indicates that while LLMs help mitigate some of the challenges faced by BLV programmers, they also introduce new forms of inaccessibility. We conducted an evaluation of five popular LLM-powered integrated development environments (IDEs), assessing their performance across a comprehensive set of programming tasks. Our findings highlight several unsupported scenarios, instances of incorrect model output, and notable limitations in interaction support for specific tasks. Through observing BLV developers as they engaged in coding activities, we uncovered key interaction barriers that go beyond model accuracy or code generation quality. This paper outlines the challenges and corresponding opportunities for improving accessibility in the context of generative AI-assisted programming. Addressing these issues can meaningfully enhance the programming experience for BLV developers. As the generative AI revolution continues to unfold, it must also address the unique burdens faced by this community.

LLM impact on BLV programming

TL;DR

This work investigates how Large Language Models (LLMs) and GenAI affect blind and low-vision (BLV) developers in programming. Through a literature review and a dual evaluation—an empirical assessment of five LLM-powered IDEs across core coding tasks and a user study with five BLV programmers—the authors identify both potential benefits and new accessibility barriers. They report inconsistent outputs across IDEs, unresolved workflow fragmentation, and AI hallucinations that complicate debugging and verification for BLV users. The findings highlight a need for accessibility-centered design, standardized AI outputs, and closer collaboration with BLV users to ensure that generative AI tools meaningfully improve programming experiences for the BLV community.

Abstract

Large Language Models (LLMs) are rapidly becoming integral to a wide range of tools, tasks, and problem-solving processes, especially in software development. Originally designed for natural language processing tasks such as text generation, LLMs are increasingly being used to assist both professionals and students in writing code. This growing reliance on LLM-based tools is reshaping programming workflows and task execution. In this study, we explore the impact of these technologies on blind and low-vision (BLV) developers. Our review of existing literature indicates that while LLMs help mitigate some of the challenges faced by BLV programmers, they also introduce new forms of inaccessibility. We conducted an evaluation of five popular LLM-powered integrated development environments (IDEs), assessing their performance across a comprehensive set of programming tasks. Our findings highlight several unsupported scenarios, instances of incorrect model output, and notable limitations in interaction support for specific tasks. Through observing BLV developers as they engaged in coding activities, we uncovered key interaction barriers that go beyond model accuracy or code generation quality. This paper outlines the challenges and corresponding opportunities for improving accessibility in the context of generative AI-assisted programming. Addressing these issues can meaningfully enhance the programming experience for BLV developers. As the generative AI revolution continues to unfold, it must also address the unique burdens faced by this community.

Paper Structure

This paper contains 11 sections, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Efforts
  3. Can LLMs Provide Solutions?
  4. Problems Introduced with LLMs
  5. Second-Hand Evaluation of LLM-powered tools
  6. "Key" Observations
  7. Problems identified by BLV people while completing tasks.
  8. User Study
  9. Participant Details
  10. "Key" Observations
  11. Conclusion

Figures (6)

  • Figure 1: LLMs-integrated tool reasons for lack of support
  • Figure 2: Examples of inaccessibility of LLM response; details provided as well.
  • Figure 3: Different LLM-enabled IDE's response to the same prompt
  • Figure 4: IPO Chart for the first program given to the participants
  • Figure 5: ChatGPT Hallucination for Wordle-like code
  • ...and 1 more figures