Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models

Guo Chen, Zhiqi Li, Shihao Wang, Jindong Jiang, Yicheng Liu, Lidong Lu, De-An Huang, Wonmin Byeon, Matthieu Le, Tuomas Rintamaki, Tyler Poon, Max Ehrlich, Tuomas Rintamaki, Tyler Poon, Tong Lu, Limin Wang, Bryan Catanzaro, Jan Kautz, Andrew Tao, Zhiding Yu, Guilin Liu

TL;DR

Eagle 2.5 introduces native long-context vision-language models that scale effectively with input length through information-first sampling and progressive post-training. The data pipeline combines open-source long-context data with Eagle-Video-110K, featuring dual-story/top-down and clip-level annotations to support narrative and spatiotemporal understanding. Empirical results show strong performance on long-video benchmarks and high-resolution image tasks, with Eagle 2.5-8B rivaling larger models and commercial systems on several metrics. The work also delivers extensive engineering optimizations for long-context training and inference, establishing a practical pathway for frontier VLMs in complex real-world scenarios.

Abstract

We introduce Eagle 2.5, a family of frontier vision-language models (VLMs) for long-context multimodal learning. Our work addresses the challenges in long video comprehension and high-resolution image understanding, introducing a generalist framework for both tasks. The proposed training framework incorporates Automatic Degrade Sampling and Image Area Preservation, two techniques that preserve contextual integrity and visual details. The framework also includes numerous efficiency optimizations in the pipeline for long-context data training. Finally, we propose Eagle-Video-110K, a novel dataset that integrates both story-level and clip-level annotations, facilitating long-video understanding. Eagle 2.5 demonstrates substantial improvements on long-context multimodal benchmarks, providing a robust solution to the limitations of existing VLMs. Notably, our best model Eagle 2.5-8B achieves 72.4% on Video-MME with 512 input frames, matching the results of top-tier commercial model such as GPT-4o and large-scale open-source models like Qwen2.5-VL-72B and InternVL2.5-78B.

Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models

TL;DR

Eagle 2.5 introduces native long-context vision-language models that scale effectively with input length through information-first sampling and progressive post-training. The data pipeline combines open-source long-context data with Eagle-Video-110K, featuring dual-story/top-down and clip-level annotations to support narrative and spatiotemporal understanding. Empirical results show strong performance on long-video benchmarks and high-resolution image tasks, with Eagle 2.5-8B rivaling larger models and commercial systems on several metrics. The work also delivers extensive engineering optimizations for long-context training and inference, establishing a practical pathway for frontier VLMs in complex real-world scenarios.

Abstract

We introduce Eagle 2.5, a family of frontier vision-language models (VLMs) for long-context multimodal learning. Our work addresses the challenges in long video comprehension and high-resolution image understanding, introducing a generalist framework for both tasks. The proposed training framework incorporates Automatic Degrade Sampling and Image Area Preservation, two techniques that preserve contextual integrity and visual details. The framework also includes numerous efficiency optimizations in the pipeline for long-context data training. Finally, we propose Eagle-Video-110K, a novel dataset that integrates both story-level and clip-level annotations, facilitating long-video understanding. Eagle 2.5 demonstrates substantial improvements on long-context multimodal benchmarks, providing a robust solution to the limitations of existing VLMs. Notably, our best model Eagle 2.5-8B achieves 72.4% on Video-MME with 512 input frames, matching the results of top-tier commercial model such as GPT-4o and large-scale open-source models like Qwen2.5-VL-72B and InternVL2.5-78B.

Paper Structure

This paper contains 26 sections, 4 equations, 8 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Eagle 2.5
  4. Model Architecture
  5. Training Strategy
  6. Information-First Sampling
  7. Post-Training Schedule
  8. Data Recipe
  9. Open-Source Long-Context Data
  10. Eagle-Video-110K
  11. Experiments
  12. Comparison with State-of-the-Art VLMs
  13. Ablation Studies
  14. Conclusion
  15. Acknowledgments
  16. ...and 11 more sections

Figures (8)

  • Figure 1: Performance comparison of Eagle 2.5 with leading vision-language models on the Video-MME benchmark. Eagle 2.5 demonstrates consistent improvement as the number of input frames increases.
  • Figure 2: Tiling-based general multimodal system.
  • Figure 3: Image area preservation. Compared to the tiling strategy (a) from InternVL chen2024internvl2, our method (b) effectively retains a larger portion of the original image, especially for high-resolution inputs. This ensures that more comprehensive visual information is preserved, benefiting tasks that require fine-grained details.
  • Figure 4: Comparison of video duration between open-source data and Eagle-Video-110K.
  • Figure 5: Overview of our video annotation framework combining bottom-up clip-level and top-down story-level approaches. The diagram illustrates our dual annotation strategy. In the bottom-up approach (left), short video clips are processed by GPT-4o to generate clip-level QA pairs enhanced with time anchors and textural context anchors. In the top-down approach (right), human annotators create story-level segmentations of longer videos, which are then captioned and processed by GPT-4 to generate comprehensive story-level QA pairs. This hierarchical methodology enables both fine-grained temporal understanding and high-level semantic comprehension of video content.
  • ...and 3 more figures