Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GIRAFFE: Design Choices for Extending the Context Length of Visual Language Models

Mukai Li, Lei Li, Shansan Gong, Qi Liu

TL;DR

This work addresses the challenge of extending Visual Language Model context lengths to support long-range visual reasoning across multi-image and video inputs. It introduces ETVLM as a diversified data recipe, M-RoPE++ as a targeted position-embedding extension, and hybrid-resolution training to maximize the utility of fixed-context tokens. The resulting Giraffe models achieve state-of-the-art performance among open-source long-context VLMs and are competitive with commercial systems on long-video benchmarks, while preserving strong short-context capabilities. These contributions enable scalable long-context visual understanding and practical deployment for visual agents and world-model-like tasks, with open-source code, data, and models to accelerate further research.

Abstract

Visual Language Models (VLMs) demonstrate impressive capabilities in processing multimodal inputs, yet applications such as visual agents, which require handling multiple images and high-resolution videos, demand enhanced long-range modeling. Moreover, existing open-source VLMs lack systematic exploration into extending their context length, and commercial models often provide limited details. To tackle this, we aim to establish an effective solution that enhances long context performance of VLMs while preserving their capacities in short context scenarios. Towards this goal, we make the best design choice through extensive experiment settings from data curation to context window extending and utilizing: (1) we analyze data sources and length distributions to construct ETVLM - a data recipe to balance the performance across scenarios; (2) we examine existing position extending methods, identify their limitations and propose M-RoPE++ as an enhanced approach; we also choose to solely instruction-tune the backbone with mixed-source data; (3) we discuss how to better utilize extended context windows and propose hybrid-resolution training. Built on the Qwen-VL series model, we propose Giraffe, which is effectively extended to 128K lengths. Evaluated on extensive long context VLM benchmarks such as VideoMME and Viusal Haystacks, our Giraffe achieves state-of-the-art performance among similarly sized open-source long VLMs and is competitive with commercial model GPT-4V. We will open-source the code, data, and models.

GIRAFFE: Design Choices for Extending the Context Length of Visual Language Models

TL;DR

This work addresses the challenge of extending Visual Language Model context lengths to support long-range visual reasoning across multi-image and video inputs. It introduces ETVLM as a diversified data recipe, M-RoPE++ as a targeted position-embedding extension, and hybrid-resolution training to maximize the utility of fixed-context tokens. The resulting Giraffe models achieve state-of-the-art performance among open-source long-context VLMs and are competitive with commercial systems on long-video benchmarks, while preserving strong short-context capabilities. These contributions enable scalable long-context visual understanding and practical deployment for visual agents and world-model-like tasks, with open-source code, data, and models to accelerate further research.

Abstract

Visual Language Models (VLMs) demonstrate impressive capabilities in processing multimodal inputs, yet applications such as visual agents, which require handling multiple images and high-resolution videos, demand enhanced long-range modeling. Moreover, existing open-source VLMs lack systematic exploration into extending their context length, and commercial models often provide limited details. To tackle this, we aim to establish an effective solution that enhances long context performance of VLMs while preserving their capacities in short context scenarios. Towards this goal, we make the best design choice through extensive experiment settings from data curation to context window extending and utilizing: (1) we analyze data sources and length distributions to construct ETVLM - a data recipe to balance the performance across scenarios; (2) we examine existing position extending methods, identify their limitations and propose M-RoPE++ as an enhanced approach; we also choose to solely instruction-tune the backbone with mixed-source data; (3) we discuss how to better utilize extended context windows and propose hybrid-resolution training. Built on the Qwen-VL series model, we propose Giraffe, which is effectively extended to 128K lengths. Evaluated on extensive long context VLM benchmarks such as VideoMME and Viusal Haystacks, our Giraffe achieves state-of-the-art performance among similarly sized open-source long VLMs and is competitive with commercial model GPT-4V. We will open-source the code, data, and models.

Paper Structure

This paper contains 30 sections, 6 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. How to Curate Extending Data
  3. Evaluation Tasks
  4. Extending Data Curation
  5. Data Composition
  6. Data Processing
  7. Data Recipe Exploration
  8. Data Ratio
  9. Data Length
  10. How to Extend Context Length
  11. Effective Length of VLMs
  12. Position Extending on VLM
  13. M-RoPE
  14. Position extending on M-RoPE
  15. M-RoPE++
  16. ...and 15 more sections

Figures (4)

  • Figure 1: Pipeline of extending visual language models. We collect data from text, text-image pairs, and videos. We propose M-RoPE++ in extending training and hybrid-resolution inference to enhance the model performance.
  • Figure 2: Performance of extending Qwen-VL with different data composition ratios.
  • Figure 3: Performance on Qwen-VL trained with different composition ratio of long ($>$8K) and short data.
  • Figure 4: Results on visual haystack. The x-axis shows the number of input images, and the y-axis shows the retrieval success rate. The dashed line indicates the 60% threshold for effective length.