Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Models

Tongtian Yue, Jie Cheng, Longteng Guo, Xingyuan Dai, Zijia Zhao, Xingjian He, Gang Xiong, Yisheng Lv, Jing Liu

TL;DR

A novel fine-tuning paradigm named Self-Consistency Tuning (SC-Tune), which features the syn-ergistic learning of a cyclic describer-locator system and is not only data-efficient but also exhibits gener-alizability across multiple LVLMs.

Abstract

Recent trends in Large Vision Language Models (LVLMs) research have been increasingly focusing on advancing beyond general image understanding towards more nuanced, object-level referential comprehension. In this paper, we present and delve into the self-consistency capability of LVLMs, a crucial aspect that reflects the models' ability to both generate informative captions for specific objects and subsequently utilize these captions to accurately re-identify the objects in a closed-loop process. This capability significantly mirrors the precision and reliability of fine-grained visual-language understanding. Our findings reveal that the self-consistency level of existing LVLMs falls short of expectations, posing limitations on their practical applicability and potential. To address this gap, we introduce a novel fine-tuning paradigm named Self-Consistency Tuning (SC-Tune). It features the synergistic learning of a cyclic describer-locator system. This paradigm is not only data-efficient but also exhibits generalizability across multiple LVLMs. Through extensive experiments, we demonstrate that SC-Tune significantly elevates performance across a spectrum of object-level vision-language benchmarks and maintains competitive or improved performance on image-level vision-language benchmarks. Both our model and code will be publicly available at https://github.com/ivattyue/SC-Tune.

SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Models

TL;DR

A novel fine-tuning paradigm named Self-Consistency Tuning (SC-Tune), which features the syn-ergistic learning of a cyclic describer-locator system and is not only data-efficient but also exhibits gener-alizability across multiple LVLMs.

Abstract

Recent trends in Large Vision Language Models (LVLMs) research have been increasingly focusing on advancing beyond general image understanding towards more nuanced, object-level referential comprehension. In this paper, we present and delve into the self-consistency capability of LVLMs, a crucial aspect that reflects the models' ability to both generate informative captions for specific objects and subsequently utilize these captions to accurately re-identify the objects in a closed-loop process. This capability significantly mirrors the precision and reliability of fine-grained visual-language understanding. Our findings reveal that the self-consistency level of existing LVLMs falls short of expectations, posing limitations on their practical applicability and potential. To address this gap, we introduce a novel fine-tuning paradigm named Self-Consistency Tuning (SC-Tune). It features the synergistic learning of a cyclic describer-locator system. This paradigm is not only data-efficient but also exhibits generalizability across multiple LVLMs. Through extensive experiments, we demonstrate that SC-Tune significantly elevates performance across a spectrum of object-level vision-language benchmarks and maintains competitive or improved performance on image-level vision-language benchmarks. Both our model and code will be publicly available at https://github.com/ivattyue/SC-Tune.
Paper Structure (33 sections, 4 equations, 5 figures, 11 tables)

This paper contains 33 sections, 4 equations, 5 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Referential Comprehension of LVLMs.
  4. Reinforcement Learning in LLMs.
  5. Self-Consistency Tuning
  6. LVLM Architecture
  7. Overview of SC-Tune
  8. Describer Training
  9. PPO Loss.
  10. Reward.
  11. Value-Free Advantage.
  12. Locator Training
  13. Experiments
  14. Experimental Settings
  15. Baseline LVLMs.
  16. ...and 18 more sections

Figures (5)

  • Figure 1: Upper: two examples demonstrating the shortfall in self-consistent referential comprehension of LVLMs, which are attributed to limited grounding (the first example) and limited captioning (the second example) capabilities, respectively. Lower: We use Pr@0.5 as the self-consistency evaluation metric. We consider it to be correct when the IoU between prediction bbox and the groung truth is greater than 0.5. The self-consistency levels of different LVLMs show pronounced performance gap between in-domain (RefCOCO) and out-of-domain (Object365 and OpenImages) datasets.
  • Figure 2: Architecture of object-level LVLMs. They mainly comprises three parts: the visual backbone for feature extraction, the vision-language connector for semantic alignment and a LLM. It has preliminary REC and REG capabilities by generating and understanding coordinates represented by text.
  • Figure 3: An illustration of our SC-Tune framework. It mainly comprises describer training cycle and locator training cycle. Each cycle employs a bbox-caption-bbox pipeline with respective loss function designed for fostering the self-consistent referential comprehension capability of object-level LVLMs. Training alternates between these two cycles, with parameter synchronization post each training cycle.
  • Figure 4: Case study to demonstrate the enhanced REG capability after SC-Tune.
  • Figure 5: Case study to demonstrate the enhanced REC capability after SC-Tune.