Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

H2OVL-Mississippi Vision Language Models Technical Report

Shaikat Galib, Shanshan Wang, Guanshuo Xu, Pascal Pfeiffer, Ryan Chesler, Mark Landry, Sri Satish Ambati

TL;DR

<3-5 sentence high-level summary> H2OVL-Mississippi introduces two compact vision-language models (0.8B and 2B) trained on 37M image–text pairs to enable efficient, on-device multimodal understanding for documents and images. The architecture leverages a ViT-MLP-LLM pipeline with dynamic tiling, pixel-shuffle token reduction, and MSAC for the 2B variant, coupled with a targeted pre-training and OCR-focused fine-tuning regime. Evaluations show the 0.8B model achieves state-of-the-art OCR text recognition, while the 2B model attains strong general multimodal performance and robust document-understanding capabilities, often exceeding similarly sized rivals on OCR tasks. The work emphasizes open-source accessibility and outlines future plans for multilingual support, additional modalities, and larger scales to broaden applicability on edge devices and in enterprise contexts.

Abstract

Smaller vision-language models (VLMs) are becoming increasingly important for privacy-focused, on-device applications due to their ability to run efficiently on consumer hardware for processing enterprise commercial documents and images. These models require strong language understanding and visual capabilities to enhance human-machine interaction. To address this need, we present H2OVL-Mississippi, a pair of small VLMs trained on 37 million image-text pairs using 240 hours of compute on 8 x H100 GPUs. H2OVL-Mississippi-0.8B is a tiny model with 0.8 billion parameters that specializes in text recognition, achieving state of the art performance on the Text Recognition portion of OCRBench and surpassing much larger models in this area. Additionally, we are releasing H2OVL-Mississippi-2B, a 2 billion parameter model for general use cases, exhibiting highly competitive metrics across various academic benchmarks. Both models build upon our prior work with H2O-Danube language models, extending their capabilities into the visual domain. We release them under the Apache 2.0 license, making VLMs accessible to everyone, democratizing document AI and visual LLMs.

H2OVL-Mississippi Vision Language Models Technical Report

TL;DR

<3-5 sentence high-level summary> H2OVL-Mississippi introduces two compact vision-language models (0.8B and 2B) trained on 37M image–text pairs to enable efficient, on-device multimodal understanding for documents and images. The architecture leverages a ViT-MLP-LLM pipeline with dynamic tiling, pixel-shuffle token reduction, and MSAC for the 2B variant, coupled with a targeted pre-training and OCR-focused fine-tuning regime. Evaluations show the 0.8B model achieves state-of-the-art OCR text recognition, while the 2B model attains strong general multimodal performance and robust document-understanding capabilities, often exceeding similarly sized rivals on OCR tasks. The work emphasizes open-source accessibility and outlines future plans for multilingual support, additional modalities, and larger scales to broaden applicability on edge devices and in enterprise contexts.

Abstract

Smaller vision-language models (VLMs) are becoming increasingly important for privacy-focused, on-device applications due to their ability to run efficiently on consumer hardware for processing enterprise commercial documents and images. These models require strong language understanding and visual capabilities to enhance human-machine interaction. To address this need, we present H2OVL-Mississippi, a pair of small VLMs trained on 37 million image-text pairs using 240 hours of compute on 8 x H100 GPUs. H2OVL-Mississippi-0.8B is a tiny model with 0.8 billion parameters that specializes in text recognition, achieving state of the art performance on the Text Recognition portion of OCRBench and surpassing much larger models in this area. Additionally, we are releasing H2OVL-Mississippi-2B, a 2 billion parameter model for general use cases, exhibiting highly competitive metrics across various academic benchmarks. Both models build upon our prior work with H2O-Danube language models, extending their capabilities into the visual domain. We release them under the Apache 2.0 license, making VLMs accessible to everyone, democratizing document AI and visual LLMs.

Paper Structure

This paper contains 11 sections, 6 figures, 6 tables.

Table of Contents

  1. Abstract
  2. Introduction
  3. Related Works
  4. Model Architecture
  5. Training Methodology
  6. H2OVL-Mississippi-0.8B Model
  7. H2OVL-Mississippi-2B Model
  8. Evaluation
  9. General Vision-Language benchmarks
  10. OCR and Document centric benchmarks
  11. Conclusions and Future Work

Figures (6)

  • Figure 1: H2OVL-Mississippi Model Architecture: The diagram illustrates the procedure for processing input images and text to the LLM. The input image undergoes resizing and cropping at various aspect ratios: (a) Resizing and cropping to the closest original size and aspect ratio, (b) Resizing and cropping to a different aspect ratio, and (c) Resizing the entire image to a fixed 448x448 pixels.
  • Figure 2:
  • Figure 3:
  • Figure 5:
  • Figure 6:
  • ...and 1 more figures