RelationVLM: Making Large Vision-Language Models Understand Visual Relations

TL;DR

RelationVLM introduces a relation-aware LVLM that can understand semantic relations, temporal associations, and geometric transforms across multiple images or video frames. It uses a frozen vision encoder, a trainable adapter, and a LLM, trained via a three-stage process and powered by data constructed from public datasets through GPT-4 prompts, enabling cost-efficient learning of relational reasoning without extensive new annotations. The model demonstrates strong qualitative and quantitative performance across grounding, similarity/contrast, temporal, and geometric tasks, as well as emergent in-context learning capabilities on unseen domains such as anomaly detection and medical imaging. This work advances practical LVLM capabilities toward broader downstream tasks and potential general visual understanding, while highlighting the role of data construction and efficient fine-tuning in achieving relation-aware multimodal reasoning.

Abstract

The development of Large Vision-Language Models (LVLMs) is striving to catch up with the success of Large Language Models (LLMs), yet it faces more challenges to be resolved. Very recent works enable LVLMs to localize object-level visual contents and ground text to them. Nonetheless, current LVLMs still struggle to precisely understand visual relations due to the lack of relevant data. In this work, we present RelationVLM, a large vision-language model capable of comprehending various levels and types of relations whether across multiple images or within a video. Specifically, we devise a multi-stage relation-aware training scheme and a series of corresponding data configuration strategies to bestow RelationVLM with the capabilities of understanding semantic relations, temporal associations and geometric transforms. Extensive case studies and quantitative evaluations show RelationVLM has strong capability in understanding such relations and emerges impressive in-context capability of reasoning from few-shot examples by comparison. This work fosters the advancements of LVLMs by enabling them to support a wider range of downstream applications toward artificial general intelligence.

Figures (9)

• Figure 1: In a simple two-image comparison task, our RelationVLM highlights differences using grounded bounding boxes, while other methods miss correct differences (gray highlight) or claim no differences (marked with bold 'no difference'). '< $\textit{img}_{1}$>' and '< /$\textit{img}_{1}$>' denote the second image, while '< $\textit{loc}_{440}$>< $\textit{loc}_{977}$>' refers to the bounding box's left-top and right-bottom coordinates. More details see Sec. \ref{['sec:data_construction']}.
• Figure 2: The overall framework of RelationVLM. The RelationVLM consists of a frozen vision encoder, a learnable adapter and a learnable LLM. We employ publicly available datasets and adopt GPT-4 to curate them into a dialogue form for its training.
• Figure 3: Illustration of the data construction process.
• Figure 4: Illustrative prompts for identifying and extracting the investigated visual relation from annotated datasets, corresponding to $p^{desc}$ in Eq. \ref{['eq:1']}.
• Figure 5: Examples of RelationVLM answering questions about image relations, including Semantic Relation (a)(b), Temporal Association (c), and Geometric Transformation (d). Color-highlighted text are output along with referring the same color bounding box. And the grey-highlighted text shows details relation description provided by RelationVLM.