Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning

Xianhang Li, Yanqing Liu, Haoqin Tu, Hongru Zhu, Cihang Xie

TL;DR

OpenVision presents a fully open family of vision encoders designed for multimodal learning, addressing transparency gaps in CLIP-style backbones. Built on CLIPS with Recap-DataComp-1B data, it employs a three-stage progressive-resolution pre-training pipeline and visual instruction fine-tuning within LLaVA-era frameworks. The study reveals key design insights, including the necessity of an auxiliary decoder and synthetic captions, and demonstrates competitive or superior performance to proprietary baselines across a wide range of benchmarks, from edge devices to high-capacity servers. By releasing hundreds of checkpoints (5.9M–632.1M parameters), datasets, and training recipes, OpenVision aims to standardize open, reproducible multimodal research and enable flexible deployment across diverse hardware environments.

Abstract

OpenAI's CLIP, released in early 2021, have long been the go-to choice of vision encoder for building multimodal foundation models. Although recent alternatives such as SigLIP have begun to challenge this status quo, to our knowledge none are fully open: their training data remains proprietary and/or their training recipes are not released. This paper fills this gap with OpenVision, a fully-open, cost-effective family of vision encoders that match or surpass the performance of OpenAI's CLIP when integrated into multimodal frameworks like LLaVA. OpenVision builds on existing works -- e.g., CLIPS for training framework and Recap-DataComp-1B for training data -- while revealing multiple key insights in enhancing encoder quality and showcasing practical benefits in advancing multimodal models. By releasing vision encoders spanning from 5.9M to 632.1M parameters, OpenVision offers practitioners a flexible trade-off between capacity and efficiency in building multimodal models: larger models deliver enhanced multimodal performance, while smaller versions enable lightweight, edge-ready multimodal deployments.

OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning

TL;DR

OpenVision presents a fully open family of vision encoders designed for multimodal learning, addressing transparency gaps in CLIP-style backbones. Built on CLIPS with Recap-DataComp-1B data, it employs a three-stage progressive-resolution pre-training pipeline and visual instruction fine-tuning within LLaVA-era frameworks. The study reveals key design insights, including the necessity of an auxiliary decoder and synthetic captions, and demonstrates competitive or superior performance to proprietary baselines across a wide range of benchmarks, from edge devices to high-capacity servers. By releasing hundreds of checkpoints (5.9M–632.1M parameters), datasets, and training recipes, OpenVision aims to standardize open, reproducible multimodal research and enable flexible deployment across diverse hardware environments.

Abstract

OpenAI's CLIP, released in early 2021, have long been the go-to choice of vision encoder for building multimodal foundation models. Although recent alternatives such as SigLIP have begun to challenge this status quo, to our knowledge none are fully open: their training data remains proprietary and/or their training recipes are not released. This paper fills this gap with OpenVision, a fully-open, cost-effective family of vision encoders that match or surpass the performance of OpenAI's CLIP when integrated into multimodal frameworks like LLaVA. OpenVision builds on existing works -- e.g., CLIPS for training framework and Recap-DataComp-1B for training data -- while revealing multiple key insights in enhancing encoder quality and showcasing practical benefits in advancing multimodal models. By releasing vision encoders spanning from 5.9M to 632.1M parameters, OpenVision offers practitioners a flexible trade-off between capacity and efficiency in building multimodal models: larger models deliver enhanced multimodal performance, while smaller versions enable lightweight, edge-ready multimodal deployments.
Paper Structure (39 sections, 3 figures, 12 tables)

This paper contains 39 sections, 3 figures, 12 tables.

Table of Contents

  1. Introduction
  2. OpenVision Training and Evaluation
  3. Vision Encoder Pre-training
  4. Training Stages and Resolution.
  5. Training Details.
  6. Visual Instruction Fine-tuning and Evaluation
  7. 1. LLaVA-1.5 liu2023improved.
  8. 2. Open-LLaVA-Next chen2024open.
  9. Evaluation benchmarks.
  10. Main Results
  11. OpenVision vs. Proprietary
  12. More OpenVision Variants
  13. Scale Up for Superior Multimodal Performance.
  14. Scale Down for Resource-Limited Scenarios.
  15. Variable Patch Sizes.
  16. ...and 24 more sections

Figures (3)

  • Figure 1: The top table compares our OpenVision series to OpenAI's CLIP and Google's SigLIP. The bottom figure showcases that OpenVision attain competitive or even superior multimodal performance than OpenAI's CLIP and Google's SigLIP.
  • Figure 2: Ablations on the impact of an auxiliary decoder and synthetic captions. Results show that both contribute to better performance across multimodal benchmarks. We present performance gaps between different variants and our setting.
  • Figure 3: Comparison of training time and average multimodal performance between our OpenVision and OpenAI-CLIP on both LLaVA-1.5 and LLaVA-Next. Larger markers correspond to vision encoders with higher input resolutions. As a fully open and cost-effective vision encoder, OpenVision achieves higher performance with significantly less pre-training time.