Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UVLM: A Universal Vision-Language Model Loader for Reproducible Multimodal Benchmarking

Joan Perez, Giovanni Fusco

Abstract

Vision-Language Models (VLMs) have emerged as powerful tools for image understanding tasks, yet their practical deployment remains hindered by significant architectural heterogeneity across model families. This paper introduces UVLM (Universal Vision-Language Model Loader), a Google Colab-based framework that provides a unified interface for loading, configuring, and benchmarking multiple VLM architectures on custom image analysis tasks. UVLM currently supports two major model families -- LLaVA-NeXT and Qwen2.5-VL -- which differ fundamentally in their vision encoding, tokenization, and decoding strategies. The framework abstracts these differences behind a single inference function, enabling researchers to compare models using identical prompts and evaluation protocols. Key features include a multi-task prompt builder with support for four response types (numeric, category, boolean, text), a consensus validation mechanism based on majority voting across repeated inferences, a flexible token budget (up to 1,500 tokens) enabling users to design custom reasoning strategies through prompt engineering, and a built-in chain-of-thought reference mode for benchmarking. UVLM is designed for reproducibility, accessibility, and extensibility and as such is freely deployable on Google Colab using consumer-grade GPU resources. The paper also presents the first benchmarking of different VLMs on tasks of increasing reasoning complexity using a corpus of 120 street-view images.

UVLM: A Universal Vision-Language Model Loader for Reproducible Multimodal Benchmarking

Abstract

Vision-Language Models (VLMs) have emerged as powerful tools for image understanding tasks, yet their practical deployment remains hindered by significant architectural heterogeneity across model families. This paper introduces UVLM (Universal Vision-Language Model Loader), a Google Colab-based framework that provides a unified interface for loading, configuring, and benchmarking multiple VLM architectures on custom image analysis tasks. UVLM currently supports two major model families -- LLaVA-NeXT and Qwen2.5-VL -- which differ fundamentally in their vision encoding, tokenization, and decoding strategies. The framework abstracts these differences behind a single inference function, enabling researchers to compare models using identical prompts and evaluation protocols. Key features include a multi-task prompt builder with support for four response types (numeric, category, boolean, text), a consensus validation mechanism based on majority voting across repeated inferences, a flexible token budget (up to 1,500 tokens) enabling users to design custom reasoning strategies through prompt engineering, and a built-in chain-of-thought reference mode for benchmarking. UVLM is designed for reproducibility, accessibility, and extensibility and as such is freely deployable on Google Colab using consumer-grade GPU resources. The paper also presents the first benchmarking of different VLMs on tasks of increasing reasoning complexity using a corpus of 120 street-view images.
Paper Structure (26 sections, 1 equation, 3 figures, 12 tables)

This paper contains 26 sections, 1 equation, 3 figures, 12 tables.

Table of Contents

  1. Motivation and significance
  2. Software description
  3. Software architecture
  4. Dual-backend inference
  5. Prompt engineering and reasoning support
  6. Consensus validation
  7. Supported models
  8. Benchmark
  9. Dataset
  10. Experimental protocol
  11. Evaluation metrics
  12. Results
  13. Best overall performance.
  14. Model size vs. performance.
  15. Effect of advanced reasoning.
  16. ...and 11 more sections

Figures (3)

  • Figure 1: UVLM software architecture. The notebook is organized into three sequential blocks. Block 1 loads the selected VLM and configures hardware settings, Block 2 defines the analysis tasks through an interactive prompt builder and Block 3 processes all images in a folder and writes results to CSV.
  • Figure 2: Comparative diagram of the dual-backend inference pipelines in UVLM. Both pipelines receive the same image--prompt pair and converge at the unified response parser.
  • Figure 3: Qwen2.5-VL-32B outputs (standard and reasoning) as compared to human annotations for task 1, 2 and 4.