Visual Prompting via Image Inpainting

TL;DR

This work introduces visual prompting via image inpainting, where a grid-like visual prompt containing task exemplars and a query is completed by a high-capacity inpainting backbone to perform downstream image-to-image tasks without fine-tuning. The authors train MAE-VQGAN on a large unlabeled dataset of arXiv figures and show that, with the right data, simple inpainting can support foreground segmentation, object detection, and colorization in a task-agnostic fashion. They provide extensive analyses, including prompt engineering, prompt ensembling, and synthetic data studies, to demonstrate the method's strengths and limitations. The Computer Vision Figures Dataset is released to enable this prompting paradigm, highlighting the potential of unlabeled figure data to teach cross-task visual reasoning. Overall, the paper suggests a data-centric route to task-general vision prompting and motivates further research into the representations learned by inpainting models.

Abstract

How does one adapt a pre-trained visual model to novel downstream tasks without task-specific finetuning or any model modification? Inspired by prompting in NLP, this paper investigates visual prompting: given input-output image example(s) of a new task at test time and a new input image, the goal is to automatically produce the output image, consistent with the given examples. We show that posing this problem as simple image inpainting - literally just filling in a hole in a concatenated visual prompt image - turns out to be surprisingly effective, provided that the inpainting algorithm has been trained on the right data. We train masked auto-encoders on a new dataset that we curated - 88k unlabeled figures from academic papers sources on Arxiv. We apply visual prompting to these pretrained models and demonstrate results on various downstream image-to-image tasks, including foreground segmentation, single object detection, colorization, edge detection, etc.

Figures (15)

• Figure 1: Visual prompting via Image Inpainting.Top: Prompting Image Inpainting Models. Given input-output example(s) $(x_1, y_1)$ and image query $x_q$, we construct a grid-like single image called a visual prompt$x_{vp}$. The visual prompt is composed of the desired task example(s) and a new query image (all in green). The inpainting model goal is then to predict the masked region (red) such that it is consistent with the example(s). Bottom: an inpainting model can solve this way various computer vision tasks, given that it was trained on the right data. The model predictions are annotated in red.
• Figure 2: MAE-VQGAN Architecture. During training, an input image is patchified, masked and fed into an MAE mae. For each masked token, the decoder outputs a distribution over a pretrained VQGAN esser2021taming codebook. The model is trained using cross entropy loss.
• Figure 3: Random images from our Computer Vision Figures dataset. We curated a dataset of 88k unlabeled figures from Computer Vision academic papers. During training, we randomly sample crops from these figures, without any additional parsing.
• Figure 4: Visual prompting prediction examples. Each visual prompt was fed to an MAE-VQGAN model trained on the Figures dataset. For each visual prompt, the result is marked in red.
• Figure 5: Synthetic data study results. MAE-VQGAN predictions are annotated with a red square.