Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can We Predict Performance of Large Models across Vision-Language Tasks?

Qinyu Zhao, Ming Xu, Kartik Gupta, Akshay Asthana, Liang Zheng, Stephen Gould

TL;DR

This work tackles the high cost of evaluating large vision-language models by formulating performance prediction as a probabilistic matrix factorization task on a sparse model-by-dataset score matrix ${\boldsymbol{R}}$. It advances PMF with MCMC to quantify uncertainty, introduces tensor-factorization for multiple metrics, and augments predictions with model/dataset profiles, including Bayesian priors with an LKJ structure. Through extensive experiments on 108 LVLMs across 176 datasets, the approach achieves accurate imputation, reliable uncertainty estimates for ordering evaluations, and improved performance under sparse data, while enabling active evaluation that prioritizes high-uncertainty predictions. The framework also demonstrates generalization to new models/datasets and shows how coresets can be combined with PMF for further efficiency. Overall, it offers a scalable method to predict LVLM performance, guiding efficient evaluation and reducing computational costs in multimodal benchmarking.

Abstract

Evaluating large vision-language models (LVLMs) is very expensive, due to high computational cost and the wide variety of tasks. The good news is that if we already have some observed performance scores, we may be able to infer unknown ones. In this study, we propose a new framework for predicting unknown performance scores based on observed ones from other LVLMs or tasks. We first formulate the performance prediction as a matrix completion task. Specifically, we construct a sparse performance matrix $\boldsymbol{R}$, where each entry $R_{mn}$ represents the performance score of the $m$-th model on the $n$-th dataset. By applying probabilistic matrix factorization (PMF) with Markov chain Monte Carlo (MCMC), we can complete the performance matrix, i.e., predict unknown scores. Additionally, we estimate the uncertainty of performance prediction based on MCMC. Practitioners can evaluate their models on untested tasks with higher uncertainty first, which quickly reduces the prediction errors. We further introduce several improvements to enhance PMF for scenarios with sparse observed performance scores. Our experiments demonstrate the accuracy of PMF in predicting unknown scores, the reliability of uncertainty estimates in ordering evaluations, and the effectiveness of our enhancements for handling sparse data.

Can We Predict Performance of Large Models across Vision-Language Tasks?

TL;DR

This work tackles the high cost of evaluating large vision-language models by formulating performance prediction as a probabilistic matrix factorization task on a sparse model-by-dataset score matrix . It advances PMF with MCMC to quantify uncertainty, introduces tensor-factorization for multiple metrics, and augments predictions with model/dataset profiles, including Bayesian priors with an LKJ structure. Through extensive experiments on 108 LVLMs across 176 datasets, the approach achieves accurate imputation, reliable uncertainty estimates for ordering evaluations, and improved performance under sparse data, while enabling active evaluation that prioritizes high-uncertainty predictions. The framework also demonstrates generalization to new models/datasets and shows how coresets can be combined with PMF for further efficiency. Overall, it offers a scalable method to predict LVLM performance, guiding efficient evaluation and reducing computational costs in multimodal benchmarking.

Abstract

Evaluating large vision-language models (LVLMs) is very expensive, due to high computational cost and the wide variety of tasks. The good news is that if we already have some observed performance scores, we may be able to infer unknown ones. In this study, we propose a new framework for predicting unknown performance scores based on observed ones from other LVLMs or tasks. We first formulate the performance prediction as a matrix completion task. Specifically, we construct a sparse performance matrix , where each entry represents the performance score of the -th model on the -th dataset. By applying probabilistic matrix factorization (PMF) with Markov chain Monte Carlo (MCMC), we can complete the performance matrix, i.e., predict unknown scores. Additionally, we estimate the uncertainty of performance prediction based on MCMC. Practitioners can evaluate their models on untested tasks with higher uncertainty first, which quickly reduces the prediction errors. We further introduce several improvements to enhance PMF for scenarios with sparse observed performance scores. Our experiments demonstrate the accuracy of PMF in predicting unknown scores, the reliability of uncertainty estimates in ordering evaluations, and the effectiveness of our enhancements for handling sparse data.

Paper Structure

This paper contains 32 sections, 5 equations, 12 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Benchmarking LVLMs
  4. Improve Evaluation Efficiency
  5. Probabilistic Matrix Factorization
  6. Modeling LVLM Performance
  7. Revisit PMF
  8. Active Evaluation
  9. Multiple Metrics
  10. Bayesian PMF
  11. Model and Dataset Profiles
  12. Experiments
  13. Evaluating Models on Benchmarks
  14. Estimating Unknown Performance
  15. Active Evaluation for LVLMs
  16. ...and 17 more sections

Figures (12)

  • Figure 1: Framework. (A) Given a sparse matrix of performance scores of LVLMs on various tasks, the goal is to estimate the missing entries. (B) A normal way is to evaluate untested model-dataset pairs one-by-one. (C) TinyBenchmarks tinybenchmarks runs models on smaller test sets and reproduce the original performance. (D) We use PMF to predict missing entries, reducing unnecessary evaluations, and rank new experiments based on uncertainty.
  • Figure 2: Graphical Models of PMF (A) and the enhanced model (B). (A) is adapted from the original paper pmf. In (B), we set the mean to $\boldsymbol{0}$ and the covariance to the identity matrix, thus omitting most of the hyper-parameters for the random variable distributions.
  • Figure 3: Performance of PMF. (A-C) PMF consistently outperforms both baselines when the test ratio is below 90% for estimating all unobserved scores (A), accuracy scores (B), and BART scores (C), with particularly strong performance at lower test ratios. (D-F) The predicted scores exhibit correlations with the ground truth at test ratios of 20% (D), 60% (E), and 90% (F). Gray dashed lines represent perfect prediction i.e., $y=x$. We subsampled 200 scores in (D-F) for visualization.
  • Figure 4: Comparison of Active Evaluation Methods. Starting with 20% of the data observed, we progressively conduct additional LVLM evaluations using three different strategies. (A) RMSE improvement demonstrate the advantage of our method compared to random evaluation. (B) Uncertainties from MCMC are correlated with the actual absolute errors.
  • Figure 5: Performance of Enhanced PTF. (A) BPTF shows minimal improvement over standard PTF when data is sufficient but proves particularly beneficial under sparse conditions. (B) Custom profiles improve performance when data is limited, though a gap remains compared to oracle profiles. (C) Ablation study on model and dataset profiles. "A $\mid$ B" represents using A for the model profile and B for the dataset profile. Custom model profiles lead to significant performance gains, while dataset profiles contribute only marginally. BPTF, Bayesian PTF; CPTF, Constrained PTF.
  • ...and 7 more figures