EditScribe: Non-Visual Image Editing with Natural Language Verification Loops

TL;DR

EditScribe addresses the accessibility gap in image editing for blind and low-vision individuals by enabling non-visual editing through natural language verification loops powered by large multimodal capabilities. The system guides users to describe the image, specify edits, performs the edits, and returns four types of verification feedback (Summary of Visual Changes, AI Judgement, Updated General Descriptions, Updated Object Descriptions) to support non-visual verification and iterative refinement. A user study with ten BLV participants demonstrates that most object-level edits are achievable non-visually, though users desire more edit actions, finer-grained control, and tailored verification feedback. The work contributes a ground-up design for non-visual visual authoring, a cross-modal grounding pipeline, and practical insights for integrating natural language verification loops into accessible image editing, with implications for future inclusive AI-assisted content creation.

Abstract

Image editing is an iterative process that requires precise visual evaluation and manipulation for the output to match the editing intent. However, current image editing tools do not provide accessible interaction nor sufficient feedback for blind and low vision individuals to achieve this level of control. To address this, we developed EditScribe, a prototype system that makes image editing accessible using natural language verification loops powered by large multimodal models. Using EditScribe, the user first comprehends the image content through initial general and object descriptions, then specifies edit actions using open-ended natural language prompts. EditScribe performs the image edit, and provides four types of verification feedback for the user to verify the performed edit, including a summary of visual changes, AI judgement, and updated general and object descriptions. The user can ask follow-up questions to clarify and probe into the edits or verification feedback, before performing another edit. In a study with ten blind or low-vision users, we found that EditScribe supported participants to perform and verify image edit actions non-visually. We observed different prompting strategies from participants, and their perceptions on the various types of verification feedback. Finally, we discuss the implications of leveraging natural language verification loops to make visual authoring non-visually accessible.

Figures (10)

• Figure 1: EditScribe user interface. (a) The images before and after the most recent edit, and an image labeled with masks and indexes for debugging purposes. (b) The accessible chat tagged with different heading levels helps users navigate a history of input prompts and verification feedback. (c) Users can input natural language prompts to perform edits or ask follow-up questions, and undo or redo edits.
• Figure 2: Detailed edit prompts and corresponding verification feedback for the Walkthrough and Session 3 in our study. Note that we only show the updated object descriptions for edited objects due to space constraint. The user can access all object descriptions if needed on the EditScribe interface, as demonstrated in Figure \ref{['fig:app']}. In (g), the object descriptions remained the same as before since the user did not specify an object to edit.
• Figure 3: Pipeline to generate initial general and object descriptions. EditScribe prompts GPT-4v gpt4v with input image to generate general descriptions, and uses visual bounding masks (by SEEM seem) and object indexes to perform Set-of-Mark Prompting to generate object descriptions.
• Figure 4: Pipeline to classify user prompts to actionable items. If the prompt is classified as a question, EditScribe prompts the prompt to GPT-4v gpt4v to answer the question. if the prompt is classified as an edit instruction, EditScribe extracts the intended edit action and the object of interest from the prompt.
• Figure 5: Pipeline to generate the four types of verification feedback. Summary of Visual changes compares the previous and edited images, while the updated general and object descriptions takes only edited images as input for prompting. AI Judgment takes both previous and edited images as input, as well as the texts, such as user prompts and previous and current general and object descriptions.