CollagePrompt: A Benchmark for Budget-Friendly Visual Recognition with GPT-4V

TL;DR

This work tackles the high cost of GPT-4V in visual recognition by introducing CollagePrompt, which bundles multiple images into a single prompt to enable one-shot inference. It builds a CollagePrompt benchmark with a large collage dataset and a GA-based Learning to Collage baseline that uses a graph-convolutional predictor to estimate collage performance and guide layout optimization. Empirical results show substantial cost reductions for 2x2 and 3x3 collages, with acceptable accuracy losses and improved arrangement quality over random baselines, validated across ImageNet-1K and 10 downstream datasets. The proposed CER and PCE metrics provide a practical framework for balancing cost-efficiency and recognition accuracy in zero-shot GPT-4V visual tasks and pave the way for broader adoption of cost-aware collage prompting in multimodal AI systems.

Abstract

Recent advancements in generative AI have suggested that by taking visual prompts, GPT-4V can demonstrate significant proficiency in visual recognition tasks. Despite its impressive capabilities, the financial cost associated with GPT-4V's inference presents a substantial barrier to its wide use. To address this challenge, we propose a budget-friendly collage prompting task that collages multiple images into a single visual prompt and makes GPT-4V perform visual recognition on several images simultaneously, thereby reducing the cost. We collect a dataset of various collage prompts to assess its performance in GPT-4V's visual recognition. Our evaluations reveal several key findings: 1) Recognition accuracy varies with different positions in the collage. 2) Grouping images of the same category together leads to better visual recognition results. 3) Incorrect labels often come from adjacent images. These findings highlight the importance of image arrangement within collage prompt. To this end, we construct a benchmark called CollagePrompt, which offers a platform for designing collage prompt to achieve more cost-effective visual recognition with GPT-4V. A baseline method derived from genetic algorithms to optimize collage layouts is proposed and two metrics are introduced to measure the efficiency of the optimized collage prompt. Our benchmark enables researchers to better optimize collage prompts, thus making GPT-4V more cost-effective in visual recognition. The code and data are available at this project page https://collageprompting.github.io/.

Figures (11)

• Figure 1: Visual recognition of GPT-4V with different prompting ways. (a): Standard Prompt takes one image as visual prompt for each GPT-4V's run. (b): Collage Prompt concatenates multiple images into one visual prompt and predicts class for all images in each inference. (c): The arrangement of images within collage prompting leads to significantly different results. Green indicates an accurate prediction while red indicates an wrong prediction.
• Figure 2: (a): The workflow of forming the collage prompt from a set of images and related indexes. For a set of images $\textbf{X}$ with two different position indexes $\textbf{I}$, we can obtain two collage prompts $\textbf{M}_1$ and $\textbf{M}_2$. Regarding $\textbf{X}$ as the node of a graph, the adjacency matrix of $\textbf{M}_1$ and $\textbf{M}_2$ can be represented as $\textbf{A}_1$ and $\textbf{A}_2$. (b): The average accuracy of collage prompts within evaluation datasets using the 'Worst,' 'Average', and 'Best' arrangement.
• Figure 3: (a) and (b) demonstrate the effect of category clustering, where placing images of the same category together increases overall recognition accuracy. (c) and (d) illustrate localization errors, where GPT-4V predicts the correct labels but outputs them to incorrect positions in the collage.
• Figure 4: Average Prediction Accuracy by Position.
• Figure 5: An overview of baseline method LCP. Starting with a set of images, index sets are randomly initialized, which forms multiple collage graphs. After predicting the accuracy of each collage graph via $G_{\theta^\ast}$, collage graphs that achieve top-$T$ accuracy are selected for crossover and mutation operations. This iterative process continues until reaching the maximum specified iteration and we can obtain the optimized arrangements.