AIris: An AI-powered Wearable Assistive Device for the Visually Impaired

TL;DR

AIris tackles the challenge of providing visually impaired users with comprehensive environmental awareness by coupling a camera-equipped wearable with an NLP-driven auditory interface. The system uses a distributed architecture where lightweight on-device processing handles privacy-sensitive tasks (e.g., face embeddings) while heavier perception tasks run on a central server, yielding real-time verbal descriptions, text extraction, and object recognition. Key contributions include modular onboard/offboard design, on-device face recognition with privacy-preserving embeddings, and a multi-task pipeline (scene description, OCR, money counting, notes, barcodes) that supports everyday independence. The prototype demonstrates promising accuracy and latency across modules and points to significant practical impact, with planned enhancements in latency reduction, ergonomics, personalization, and integration with advanced AI models. Overall, AIris represents a meaningful step toward AI-augmented assistive technology that can be upgraded with future AI advances to better support visually impaired individuals in real-world tasks.

Abstract

Assistive technologies for the visually impaired have evolved to facilitate interaction with a complex and dynamic world. In this paper, we introduce AIris, an AI-powered wearable device that provides environmental awareness and interaction capabilities to visually impaired users. AIris combines a sophisticated camera mounted on eyewear with a natural language processing interface, enabling users to receive real-time auditory descriptions of their surroundings. We have created a functional prototype system that operates effectively in real-world conditions. AIris demonstrates the ability to accurately identify objects and interpret scenes, providing users with a sense of spatial awareness previously unattainable with traditional assistive devices. The system is designed to be cost-effective and user-friendly, supporting general and specialized tasks: face recognition, scene description, text reading, object recognition, money counting, note-taking, and barcode scanning. AIris marks a transformative step, bringing AI enhancements to assistive technology, enabling rich interactions with a human-like feel.

Figures (4)

• Figure 1: AIris architecture overview: User is wearing the device, including earphones, camera and Raspberry Pi. The user asks a question, which Raspberry Pi converts to text and using Natural Language Processing it selects which module should be evoked. The camera input is transmitted to the server, where the ML model inference happens. The results are then sent back to the Raspberry Pi, where they are formatted into a sentence, converted from text to speech, and finally communicated to the user through the earphones.
• Figure 2: An example of how the Scene Description, Object Recognition and Face Recognition Modules interact with the user and the captured data. Photo by Brooke Cagle on Unsplash unsplashPhotoBrooke
• Figure 3: An example of how the Note Taking, Counting Money, Text Reading and Barcode Scanning Modules interact with the user and the captured data. Photo in the middle by kstudio on Freepik freepikFreePhoto
• Figure 4: AIris Design Process: This figure shows from left to right the evolution of the hardware for AIris. At first, we sketched out the basic idea and how the components would interact. Then we moved on to designing the eyewear in a Computer Aided Design (CAD) Program, focusing on keeping it light-weight, simple and user-friendly. Finally, we 3D printed our eyewear and assembled the whole prototype.