Can MLLMs Perform Text-to-Image In-Context Learning?

TL;DR

This work defines Text-to-Image In-Context Learning (T2I-ICL) and introduces CoBSAT, the first benchmark designed to test MLLMs on transforming textual prompts into images or image descriptions via in-context demonstrations. Through extensive evaluation of multiple image-generation and text-only MLLMs, the study finds that multimodal integration and image generation are primary bottlenecks, with image-description tasks easier for some models than image creation. Fine-tuning on CoBSAT and Chain-of-Thought prompting substantially improve T2I-ICL performance for several models, though gains are model-dependent and sometimes counterproductive for others. The results highlight the need for targeted prompt engineering, broader multimodal training, and possibly multimodal-CoT strategies to advance T2I-ICL capabilities, with practical implications for design and evaluation of future MLLMs. CoBSAT and associated code provide a foundation for ongoing research in this underexplored area of multimodal in-context learning.

Abstract

The evolution from Large Language Models (LLMs) to Multimodal Large Language Models (MLLMs) has spurred research into extending In-Context Learning (ICL) to its multimodal counterpart. Existing such studies have primarily concentrated on image-to-text ICL. However, the Text-to-Image ICL (T2I-ICL), with its unique characteristics and potential applications, remains underexplored. To address this gap, we formally define the task of T2I-ICL and present CoBSAT, the first T2I-ICL benchmark dataset, encompassing ten tasks. Utilizing our dataset to benchmark six state-of-the-art MLLMs, we uncover considerable difficulties MLLMs encounter in solving T2I-ICL. We identify the primary challenges as the inherent complexity of multimodality and image generation, and show that strategies such as fine-tuning and Chain-of-Thought prompting help to mitigate these difficulties, leading to notable improvements in performance. Our code and dataset are available at https://github.com/UW-Madison-Lee-Lab/CoBSAT.

Figures (28)

• Figure 1: Comparison of various In-Context Learning (ICL) settings. (a) Textual ICL, where both the input and output in each example are textual. (b) Visual ICL, where both input and output in each demonstration are presented as images. (c) Image-to-Text ICL (I2T-ICL), featuring images as input and texts as output in each demonstration. (d) Text-to-Image ICL (T2I-ICL, our focus), which involves text input and image output in each demonstration. T2I-ICL introduces greater complexities and presents different potential applications. The examples in (d) provide three potential applications of T2I-ICL, with the output generated using ChatGPT-4 openai2023gpt4 with DALL-E 3 betker2023dalle3 capabilities.
• Figure 2: Overview of example prompts in the CoBSAT benchmark. CoBSAT covers five themes: color, background, style, action, and texture, each with two different emphases: object-inference and attribute-inference. In object-inference tasks, the attribute (e.g., color) is directly provided in the textual input, and the model is required to infer the object (e.g., car) from the images. In other words, the latent variable (denoted as "Latent Var." in the figure) of object-inference tasks is the object. Conversely, in attribute-inference tasks, the object is specified in the text. The model is tasked with inferring the attribute from the images in the demonstrations, i.e., the attribute serves as the latent variable in attribute-inference tasks.
• Figure 3: Benchmarking pipline for MLLMs in T2I-ICL with CoBSAT. (i) For MLLMs with image generation capabilities, we feed prompts from our dataset into the MLLM under evaluation to prompt image generation. If the MLLM accurately interprets the text-image relationship in the provided demonstrations, it should produce an image of a "black chair." To verify this alignment, we employ one evaluation model, it could be either a Vision-Language Model (VLM, e.g., CLIP) or an MLLM adept at visual question answering (e.g., LLaVA). This allows us to determine whether the generated image accurately corresponds to the target label. (ii) For MLLMs that do not generate images, we modify the process by instructing the MLLMs to describe the image textually, following the same evaluation criteria as in the image generation scenario.
• Figure 4: T2I-ICL performance of MLLMs on CoBSAT with 2,4,6,8 demonstrations.
• Figure 5: Accuracy estimated by CLIP versus accuracy estimated by Gemini.