TalkPhoto: A Versatile Training-Free Conversational Assistant for Intelligent Image Editing

TL;DR

TalkPhoto tackles the training cost and rigidity of instruction-based image editing by offering a training-free, prompt-driven framework that leverages a plug-and-play function library guided by an LLM. It introduces hierarchical invocation and function simplification to enable accurate multi-function calls with lower token usage. It also presents a dedicated instructional image inpainting pipeline that combines object removal with segmentation-based masking and inpainting for high-quality results. Experiments show improved function invocation accuracy, reduced token consumption, and stronger editing quality versus state-of-the-art baselines, with a practical offline deployment and interactive UI.

Abstract

Thanks to the powerful language comprehension capabilities of Large Language Models (LLMs), existing instruction-based image editing methods have introduced Multimodal Large Language Models (MLLMs) to promote information exchange between instructions and images, ensuring the controllability and flexibility of image editing. However, these frameworks often build a multi-instruction dataset to train the model to handle multiple editing tasks, which is not only time-consuming and labor-intensive but also fails to achieve satisfactory results. In this paper, we present TalkPhoto, a versatile training-free image editing framework that facilitates precise image manipulation through conversational interaction. We instruct the open-source LLM with a specially designed prompt template to analyze user needs after receiving instructions and hierarchically invoke existing advanced editing methods, all without additional training. Moreover, we implement a plug-and-play and efficient invocation of image editing methods, allowing complex and unseen editing tasks to be integrated into the current framework, achieving stable and high-quality editing results. Extensive experiments demonstrate that our method not only provides more accurate invocation with fewer token consumption but also achieves higher editing quality across various image editing tasks.

Figures (14)

• Figure 1: Proposed versatile training-free image editing framework named TalkPhoto. Left: Only one prompt is needed for users to perform accurate image editing via TalkPhoto. Right: Some supported editing tasks and corresponding results.
• Figure 2: Overview of the TalkPhoto framework. For user input text instructions, we first combine it with a carefully designed prompt and input into the LLM➀ to generate a structured response➁, detailing the functions to be used along with an analysis and response to the user's needs. If the selected function involves sub-functions or requires additional details, the LLM is consulted a second time to refine the specific functions➂➃. The selected function is called from the function library➄ to process the image uploaded by the user. Finally, the edited image and the LLM's response are returned to the user➅.
• Figure 3: The details of the prompt template. It consists of prefix prompt, functions information, fixed output format, and few-shot examples in turn.
• Figure 4: Our pipeline for instructional image inpainting. The LLM responds twice in total. The first response determines the invocation of the object removal function. The second response outputs the category and detailed description of the removed object, which are then input into two segmentation models. The results are matched to obtain the optimal segmentation mask, which is subsequently used for inpainting to generate the final result.
• Figure 5: Intermediate results of object removal using our method. The first column is the original image, and the second column is the segmentation mask detected by the pre-trained description-based segmentation model unilseg. The third column shows the refined mask for inpainting. The fourth column is our final result obtained using the pre-trained inpainting model lama.