ObjectFinder: An Open-Vocabulary Assistive System for Interactive Object Search by Blind People

TL;DR

This paper introduces ObjectFinder, a wearable system that unifies open-vocabulary object detection (YOLO-World) with a multimodal language model (GPT-4) to support open-ended, interactive search for blind users in unfamiliar environments. The design emphasizes flexible target queries, real-time egocentric localization, and intent-driven feedback branches (navigation and scene description/open questions), implemented across three modules and validated by an exploratory study with eight blind participants against BeMyAI and Google Lookout. Findings show that ObjectFinder provides essential localization and scene-context information, enhances independence, and supports discovery of incidental targets, though users desire customization of information load and smoother hardware interaction. The work highlights practical opportunities and tensions in integrating vision- and AI-based assistive tech, offering concrete directions for future development, including richer descriptions, reliability improvements, and portable, socially acceptable hardware.

Abstract

Searching for objects in unfamiliar scenarios is a challenging task for blind people. It involves specifying the target object, detecting it, and then gathering detailed information according to the user's intent. However, existing description- and detection-based assistive technologies do not sufficiently support the multifaceted nature of interactive object search tasks. We present ObjectFinder, an open-vocabulary wearable assistive system for interactive object search by blind people. ObjectFinder allows users to query target objects using flexible wording. Once the target object is detected, it provides egocentric localization information in real-time, including distance and direction. Users can then initiate different branches to gather detailed information based on their intent towards the target object, such as navigating to it or perceiving its surroundings. ObjectFinder is powered by a seamless combination of open-vocabulary models, namely an open-vocabulary object detector and a multimodal large language model. The ObjectFinder design concept and its development were carried out in collaboration with a blind co-designer. To evaluate ObjectFinder, we conducted an exploratory user study with eight blind participants. We compared ObjectFinder to BeMyAI and Google Lookout, popular description- and detection-based assistive applications. Our findings indicate that most participants felt more independent with ObjectFinder and preferred it for object search, as it enhanced scene context gathering and navigation, and allowed for active target identification. Finally, we discuss the implications for future assistive systems to support interactive object search.

Figures (9)

• Figure 1: Martin walks into an unfamiliar office and uses an object-search system to search for a socket to charge his smartphone. (a) Martin first specifies the target to the system, which then repeats it for confirmation. (b) While scanning, candidates are detected. The socket "4 meters away at his 10 o'clock next to the trash bin" is not what he wants. (c) However, another socket "3 meters away at his 2 o'clock next to the workstation" is the desired one, as he plans to study there. (d) After confirming the target, Martin may ask for more details. In large rooms, the system should navigate him to the socket.
• Figure 2: Hardware design and components of the wearable system ObjectFinder. It incorporates a stereo camera to capture visual information about the user's surroundings, a pair of buttons, and a microphone to collect the user's commands. Simultaneously, it executes algorithms through a lightweight processor. To provide a comprehensive and immersive experience, the system delivers spatial-aware informational feedback directly to the user via bone-conduction headphones.
• Figure 3: ObjectFinder system architecture integrates five functions into three modules for interactive object search. (a) Initially, an open-vocabulary object detector, e.g. YOLO-World, is initialized with a list of objects extracted from a scenario capture, allowing the user to identify a target object. If the target is not on the list, the object detector is reinitialized. (b) The user scans the environment. If the target is detected, localization information is provided in real-time. If not, the user can trigger scene understanding to identify what exists in the scenario. (c) The user may activate a sub-branch to obtain further information based on their intent using a multimodal large language model. (d) If the user discovers other objects of interest or becomes disoriented, they can reorient themselves to locate the target.
• Figure 4: Initialization with target identification: (a) The list of detectable objects in YOLO-World is initialized with the first capture of the scenario. The target objects can be categorized into three types: (b) match, where the object matches an item in the list; (c) related, where the object is related to one item in the list (e.g., "couch" is related to "sofa" with 0.85 similiarity); and (d) unrelated, where the object does not relate to any item in the list. In cases where the object is unrelated, the list is updated by adding the target to it.
• Figure 5: Object detection and localization: Each video frame is processed by YOLO-World to detect key frames in which the confidence level of the bounding box around the target object exceeds a certain threshold. Subsequently, the segmentation map generated by the bounding box, combined with the depth map, is used to provide precise localization information, including distance and direction.