Point and Instruct: Enabling Precise Image Editing by Unifying Direct Manipulation and Text Instructions

TL;DR

The paper addresses the challenge of achieving precise image edits when text prompts alone are insufficient to disambiguate targets or specify exact locations. It introduces Point & Instruct, a web-based multimodal interface that unifies direct manipulation (bounding boxes, points) with natural-language instructions, and processes them via an LLM to produce a transformed image layout. The transformed layout guides a layout-based diffusion generation pipeline (GLIGEN and related tools) to edit the image, with optional subject-specific fine-tuning for consistency. The work demonstrates improved precision over text-only editing baselines and outlines ongoing user studies and evaluation plans to quantify usability and editing accuracy.

Abstract

Machine learning has enabled the development of powerful systems capable of editing images from natural language instructions. However, in many common scenarios it is difficult for users to specify precise image transformations with text alone. For example, in an image with several dogs, it is difficult to select a particular dog and move it to a precise location. Doing this with text alone would require a complex prompt that disambiguates the target dog and describes the destination. However, direct manipulation is well suited to visual tasks like selecting objects and specifying locations. We introduce Point and Instruct, a system for seamlessly combining familiar direct manipulation and textual instructions to enable precise image manipulation. With our system, a user can visually mark objects and locations, and reference them in textual instructions. This allows users to benefit from both the visual descriptiveness of natural language and the spatial precision of direct manipulation.

Figures (5)

• Figure 1: Point & Instruct enables users to perform precise image manipulations that are difficult to do with text alone. A user can leverage familiar direct manipulation to specify regions or objects in an image, which can be referred to in text instructions. By combining direct manipulation and natural language based editing it becomes much easier for users to perform precise edits like: moving a particular object, adding an object in a specified location, or changing the appearance of an object.
• Figure 2: Point & Instruct harnesses the power of LLMs to process a variety of instructions and leverages visual information specified by simple geometric objects. The flexibility of text prompts can be seamlessly combined with familiar GUI elements, making it simple to understand and use. In our example use-case, a user would (A) upload an existing image or write a text prompt to generate an image, (B) select object(s) with a bounding box specified through direct manipulation, (C) specify another location to move an object to with a bounding box or star, (D) click enter or a button to run the generation process, and finally (E) view the generated image.
• Figure 3: Point & Instruct casts the problem of image editing as a natural language generation task. (A) The input image and instruction are serialized into a textual form, and (B) an LLM accepts the input layout and instruction and produces a transformed layout. Finally, (C) a layout-to-image generation system is used to generate an edited image from the transformed layout.
• Figure 4: We leverage in-context learning to take advantage of the few-shot generalization capabilities of LLMs. We place a relatively small number ($\approx 15$) examples for our task in the context of an LLM. Each example contains (a) a serialized image layout, (b) a serialized instruction, (c) a chain of thought composed of multiple task-relevant questions meant to assist the LLM by providing it with additional context, and (d) an annotated layout specifying the relevant transformation. At inference time we place an input image layout and instruction after the in-context examples.
• Figure 5: Our approach enables a user to disambiguate a particular object from other similar objects, move it, and change its appearance. In contrast, text-only editing approaches like InstructPix2Pixbrooks_instructpix2pix_2023 and LLM Grounded Diffusionlian_llm-grounded_2023 fail to localize the manipulation to the correct object, despite requiring a much longer and more difficult to write edit instruction.