Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improved Alignment of Modalities in Large Vision Language Models

Kartik Jangra, Aman Kumar Singh, Yashwani Mann, Geetanjali Rathee

TL;DR

This work tackles the problem of efficiently aligning vision and language in large multimodal models by proposing a stepwise, auto-regressive training recipe that uses a compact 900-million-parameter network. It introduces an attention-masking paradigm and staged data mixtures to progressively align image and text modalities while freezing and then unfreezing the language model to protect pretrained capabilities. The method achieves competitive results on standard benchmarks (e.g., CIDEr on COCO and Flickr30k) with far less data and compute than larger models, and demonstrates promising capabilities in medical visual question answering (PathVQA). The findings suggest practical implications for edge-device deployment and broader accessibility, while highlighting avenues for extending masking strategies to interleaved datasets and improving data efficiency.

Abstract

Recent advancements in vision-language models have achieved remarkable results in making language models understand vision inputs. However, a unified approach to align these models across diverse tasks such as image captioning and visual question answering remains a challenge. Existing methods either require very big language models or very big datasets which is not efficient in utilizing existing models. This paper addresses this gap and devises a training strategy of auto-regressive vision-language models, to unify vision-language tasks like image-captioning and visual question answering. We propose four training stages for aligning the vision model with the language model, in other words, the language model is given an ability to process visual inputs. We also devise different attention masks for training transformer-based language models that improve the quality of visual features. Further, we introduce some findings, 1) the attention mask should not be applied on visual inputs, 2) the Language model converges faster on AI- generated data, 3) More work should be done in the alignment stage during the pre-training of the model, 4) the model can easily adapt to any downstream tasks like visual question answering on healthcare datasets like PathVQA. After training the model for one epoch for all the stages, it outperforms large models like VILA-13 billion models on common benchmarks like CIDEr scores on COCO and Flickr30k datasets and achieves very close scores to GIT-2 on the same dataset despite being a much smaller model trained on a much smaller dataset. All of the training is done using best practices available like multi- GPU parallel training, lower-precision training with 16-bit float numbers, faster attention (SDPA), and gradient accumulation, and completed the training within 12 hours.

Improved Alignment of Modalities in Large Vision Language Models

TL;DR

This work tackles the problem of efficiently aligning vision and language in large multimodal models by proposing a stepwise, auto-regressive training recipe that uses a compact 900-million-parameter network. It introduces an attention-masking paradigm and staged data mixtures to progressively align image and text modalities while freezing and then unfreezing the language model to protect pretrained capabilities. The method achieves competitive results on standard benchmarks (e.g., CIDEr on COCO and Flickr30k) with far less data and compute than larger models, and demonstrates promising capabilities in medical visual question answering (PathVQA). The findings suggest practical implications for edge-device deployment and broader accessibility, while highlighting avenues for extending masking strategies to interleaved datasets and improving data efficiency.

Abstract

Recent advancements in vision-language models have achieved remarkable results in making language models understand vision inputs. However, a unified approach to align these models across diverse tasks such as image captioning and visual question answering remains a challenge. Existing methods either require very big language models or very big datasets which is not efficient in utilizing existing models. This paper addresses this gap and devises a training strategy of auto-regressive vision-language models, to unify vision-language tasks like image-captioning and visual question answering. We propose four training stages for aligning the vision model with the language model, in other words, the language model is given an ability to process visual inputs. We also devise different attention masks for training transformer-based language models that improve the quality of visual features. Further, we introduce some findings, 1) the attention mask should not be applied on visual inputs, 2) the Language model converges faster on AI- generated data, 3) More work should be done in the alignment stage during the pre-training of the model, 4) the model can easily adapt to any downstream tasks like visual question answering on healthcare datasets like PathVQA. After training the model for one epoch for all the stages, it outperforms large models like VILA-13 billion models on common benchmarks like CIDEr scores on COCO and Flickr30k datasets and achieves very close scores to GIT-2 on the same dataset despite being a much smaller model trained on a much smaller dataset. All of the training is done using best practices available like multi- GPU parallel training, lower-precision training with 16-bit float numbers, faster attention (SDPA), and gradient accumulation, and completed the training within 12 hours.

Paper Structure

This paper contains 19 sections, 4 equations, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Common components used in designing the vision language models are:
  3. Research Gaps
  4. Contribution of paper
  5. Related Work
  6. Proposed Method
  7. Model Architecture
  8. Attention Mask
  9. Dataset Mixture
  10. Training Stages
  11. Training Methods
  12. Performance Analysis
  13. Training Platform and specification
  14. Training Hyperparameters
  15. Results and Discussion
  16. ...and 4 more sections

Figures (5)

  • Figure 1: Model performance on different datasets.
  • Figure 2: The model architecture, it consists of a vision encoder, a multi-modal projector and a language model.
  • Figure 4: Training loss curves for different stages of training.
  • Figure 5: Training loss comparisons for different datasets.
  • Figure 6: Visualization of attention masks for interleaved datasets.