VEGA: Learning Interleaved Image-Text Comprehension in Vision-Language Large Models

TL;DR

This work introduces Interleaved Image-Text Comprehension (IITC), a challenging multimodal task in which models must locate the relevant image and text within long, interleaved content and answer with the appropriate image index. It contributes VEGA, a dataset with IITC and Image-Text Association (ITA) subsets built from SciGraphQA, supporting long contexts up to 8 images and 8k tokens. A multi-task, multi-scale training approach is applied to fine-tune Qwen-VL-Chat, achieving an image association accuracy of 85.8% and a Rouge score of 0.508 on VEGA, while revealing gaps in both open-source and proprietary models. The results establish a strong baseline for IITC/ITA in real-world scientific document comprehension and set directions for future enhancements in long-context multimodal reasoning.

Abstract

The swift progress of Multi-modal Large Models (MLLMs) has showcased their impressive ability to tackle tasks blending vision and language. Yet, most current models and benchmarks cater to scenarios with a narrow scope of visual and textual contexts. These models often fall short when faced with complex comprehension tasks, which involve navigating through a plethora of irrelevant and potentially misleading information in both text and image forms. To bridge this gap, we introduce a new, more demanding task known as Interleaved Image-Text Comprehension (IITC). This task challenges models to discern and disregard superfluous elements in both images and text to accurately answer questions and to follow intricate instructions to pinpoint the relevant image. In support of this task, we further craft a new VEGA dataset, tailored for the IITC task on scientific content, and devised a subtask, Image-Text Association (ITA), to refine image-text correlation skills. Our evaluation of four leading closed-source models, as well as various open-source models using VEGA, underscores the rigorous nature of IITC. Even the most advanced models, such as Gemini-1.5-pro and GPT4V, only achieved modest success. By employing a multi-task, multi-scale post-training strategy, we have set a robust baseline for MLLMs on the IITC task, attaining an $85.8\%$ accuracy rate in image association and a $0.508$ Rouge score. These results validate the effectiveness of our dataset in improving MLLMs capabilities for nuanced image-text comprehension.

Figures (14)

• Figure 1: Comparison between existing VQA tasks and our IITC task. Left: The input for existing VQA tasks only incorporates a limited amount of image and text data, which is highly relevant to the question. Right: The input for the IITC task contains longer images and text information, which includes redundant and misleading data. The model needs to specify the reference image when providing an answer.
• Figure 2: The task definition of IITC and ITA tasks. (a) The IITC task takes long interleaved image-text content as input and requires the model to specify the image it refers to in its response. (b) The ITA task takes shuffled images and text segments from different articles as input and requires the model to output the relationship between the text and the images. <Text *> and <Image *> represent a text segment and an image, respectively. They are both tokenized and fed into the model along with the task prompt and the question.
• Figure 3: The construction process of the VEGA dataset. We use the SciGraphQA dataset as a foundation, to which we add more images and text to its context, modify questions and answers that lack clear image references, and incorporates references to meet the requirements of the IITC task.
• Figure 4: The distribution of the number of images and tokens in the IITC subset of the VEGA dataset. The number of tokens for each image is 256.
• Figure 5: The composition of the VEGA Dataset.