On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning

TL;DR

This study aims to redefine the design of vision-language models by identifying key components and creating efficient models with constrained inference costs by strategically formulating datasets, optimizing vision modules, and enhancing supervision techniques.

Abstract

Recent advancements in language and vision assistants have showcased impressive capabilities but suffer from a lack of transparency, limiting broader research and reproducibility. While open-source models handle general image tasks effectively, they face challenges with the high computational demands of complex visually-situated text understanding. Such tasks often require increased token inputs and large vision modules to harness high-resolution information. Striking a balance between model size and data importance remains an open question. This study aims to redefine the design of vision-language models by identifying key components and creating efficient models with constrained inference costs. By strategically formulating datasets, optimizing vision modules, and enhancing supervision techniques, we achieve significant improvements in inference throughput while maintaining high performance. Extensive experiments across models ranging from 160M to 13B parameters offer insights into model optimization. We will fully open-source our codebase, models, and datasets at https://github.com/naver-ai/elva.

Figures (11)

• Figure 1: Graphical comparison illustrating average score against latency and memory consumption for various models. Scores are derived from eight benchmarks: DocVQA mathew2021docvqa, ChartQA masry-etal-2022-chartqa, InfographicVQA Mathew_2022_WACV, SEED-IMG li2023seed, SEED-2-Plus li2024seed2plus, MMStar chen2024we, ScienceQA lu2022learn, and HallusionBench Guan_2024_CVPR. See Section \ref{['sec:problem_and_improvement']} for benchmark details. Elva excels with high performance, reduced latency, and lower memory usage. Right: Performance improvements from LLaVA to Elva at the 7B scale, achieved through strategies in Section \ref{['sec:remedies']}.
• Figure 2: Training pipeline consists of two stages. Alignment of visual and textual features through the MLP, followed by joint training of the LM and the MLP.
• Figure 3: Performance of various vision encoder configurations at 1B, 7B, and 13B. Average scores for each configuration (C1 to C7) across 8 benchmarks.
• Figure 4: Impact of RR-Prompt with a 10% dataset subset. Results demonstrate effects during training.
• Figure 5: Latency comparison across multiple benchmarks.Elva delivers promising results.