How Much Data Is Enough? Uniform Convergence Bounds for Generative & Vision-Language Models under Low-Dimensional Structure

TL;DR

This work addresses the problem of when generative and vision-language models can achieve uniformly accurate and well-calibrated predictions in data-limited, biomedical contexts. It develops a finite-sample uniform convergence framework for VLM-induced classifiers by leveraging Lipschitz stability with respect to a low-dimensional prompt embedding and the spectral decay of embeddings, yielding bounds that scale with intrinsic dimension $d$ and eigenvalues $\lambda_i$ rather than ambient dimension $D$. The main contributions include a model-agnostic uniform convergence lemma based on $\varepsilon$-covers, spectrum-aware sample complexity bounds, and concrete implications for data planning and calibration evaluation in medical settings, highlighting why average calibration metrics may miss worst-case miscalibration. Together, these results clarify how structure in semantic prompt spaces enables reliable, uniformly calibrated predictions with practical data sizes, informing prompting strategies and data collection in clinical AI deployments.

Abstract

Modern generative and vision-language models (VLMs) are increasingly used in scientific and medical decision support, where predicted probabilities must be both accurate and well calibrated. Despite strong empirical results with moderate data, it remains unclear when such predictions generalize uniformly across inputs, classes, or subpopulations, rather than only on average-a critical issue in biomedicine, where rare conditions and specific groups can exhibit large errors even when overall loss is low. We study this question from a finite-sample perspective and ask: under what structural assumptions can generative and VLM-based predictors achieve uniformly accurate and calibrated behavior with practical sample sizes? Rather than analyzing arbitrary parameterizations, we focus on induced families of classifiers obtained by varying prompts or semantic embeddings within a restricted representation space. When model outputs depend smoothly on a low-dimensional semantic representation-an assumption supported by spectral structure in text and joint image-text embeddings-classical uniform convergence tools yield meaningful non-asymptotic guarantees. Our main results give finite-sample uniform convergence bounds for accuracy and calibration functionals of VLM-induced classifiers under Lipschitz stability with respect to prompt embeddings. The implied sample complexity depends on intrinsic/effective dimension, not ambient embedding dimension, and we further derive spectrum-dependent bounds that make explicit how eigenvalue decay governs data requirements. We conclude with implications for data-limited biomedical modeling, including when current dataset sizes can support uniformly reliable predictions and why average calibration metrics may miss worst-case miscalibration.

Figures (2)

• Figure 1: Geometrical intuition for uniform convergence. The horizontal axis denotes the prompt embedding $p$, and the vertical axis denotes the classifier score $f(p)$. The empirical function $f_n(p)$ approximates the true function $f(p)$ to within $\varepsilon$ uniformly across $p$. Because $f$ is Lipschitz with constant $L$, changes in the score are bounded by distance in embedding space, forming a Lipschitz tube around $f$. Covering the prompt space with balls of radius $\rho = O(\varepsilon/L)$ ensures that controlling error at a finite set of representative prompts suffices to control the error everywhere.
• Figure 2: Class probability outputs of a vision--language model (VLM) as a function of the prompt embedding. The figure illustrates how predicted class probabilities vary smoothly across the semantic embedding space (shown in a 2D projection for visualization), enabling uniform control of accuracy and calibration under Lipschitz assumptions.