Lever LM: Configuring In-Context Sequence to Lever Large Vision Language Models

TL;DR

Experiments show that these ICD sequences can improve the ICL performance of two LVLMs compared with some strong baselines in Visual Question Answering and Image Captioning, validating that Lever-LM can really capture the statistical patterns for levering LVLMs.

Abstract

As Archimedes famously said, ``Give me a lever long enough and a fulcrum on which to place it, and I shall move the world'', in this study, we propose to use a tiny Language Model (LM), \eg, a Transformer with 67M parameters, to lever much larger Vision-Language Models (LVLMs) with 9B parameters. Specifically, we use this tiny \textbf{Lever-LM} to configure effective in-context demonstration (ICD) sequences to improve the In-Context Learinng (ICL) performance of LVLMs. Previous studies show that diverse ICD configurations like the selection and ordering of the demonstrations heavily affect the ICL performance, highlighting the significance of configuring effective ICD sequences. Motivated by this and by re-considering the the process of configuring ICD sequence, we find this is a mirror process of human sentence composition and further assume that effective ICD configurations may contain internal statistical patterns that can be captured by Lever-LM. Then a dataset with effective ICD sequences is constructed to train Lever-LM. After training, given novel queries, new ICD sequences are configured by the trained Lever-LM to solve vision-language tasks through ICL. Experiments show that these ICD sequences can improve the ICL performance of two LVLMs compared with some strong baselines in Visual Question Answering and Image Captioning, validating that Lever-LM can really capture the statistical patterns for levering LVLMs. The code is available at \url{https://github.com/ForJadeForest/Lever-LM}.

Figures (5)

• Figure 1: (a) The traditional ICD configuration methods separately select and order the ICDs, leading to sub-optimal ICL performance. (b) Our Lever-LM enables the step-by-step generation of ICD configurations and simultaneously considers the selection of ICDs and the ordering of ICD sequences.
• Figure 2: (a): The pipeline of constructing $\mathcal{D}_{\mathcal{M}}$. Darker color of $S^K_{i,j}$ indicates a higher score given by Eq. \ref{['eq:constrcut-dataset']}. (b): Top: Lever-LM is a two-layer Transformer. Bottom: Each input embeddings is the sum of the random initialized learnable embeddings, the image and text embeddings extracted by CLIP. The dotted block means that some tasks do not exist the text input, e.g., IC.
• Figure 3: Visualizations of diverse ICDs configurations, where the first and the last ICDs are given due to space limitation. We can find that Lever-LM use more diverse ICDs and thus not lead to short-cut inference.
• Figure 4: 8-shot ICD configurations visualizations of IC Fixed Set.
• Figure 5: 8-shot ICD configurations visualizations of VQA Fixed Set.