Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Function-Space Empirical Bayes Regularisation with Large Vision-Language Model Priors

Pengcheng Hao, Huaze Tang, Ercan Engin Kuruoglu, Wenbo Ding

TL;DR

The paper tackles uncertainty quantification for high-dimensional deep learning by coupling function-space empirical Bayes with large vision-language models. It introduces VLM-FS-EB, which generates semantically meaningful context points via VLMs and forms expressive function-space priors from frozen VLM embeddings, enabling robust posterior inference with MC dropout. Empirical results across multiple datasets show improved predictive performance and, crucially, superior out-of-distribution detection, particularly in data-scarce regimes. This approach reduces reliance on task-specific pretraining data while leveraging foundation-model representations to achieve reliable uncertainty estimates in safety-critical settings.

Abstract

Bayesian deep learning (BDL) provides a principled framework for reliable uncertainty quantification by combining deep neural networks with Bayesian inference. A central challenge in BDL lies in the design of informative prior distributions that scale effectively to high-dimensional data. Recent functional variational inference (VI) approaches address this issue by imposing priors directly in function space; however, most existing methods rely on Gaussian process (GP) priors, whose expressiveness and generalisation capabilities become limited in high-dimensional regimes. In this work, we propose VLM-FS-EB, a novel function-space empirical Bayes regularisation framework, leveraging large vision-language models (VLMs) to generates semantically meaningful context points. These synthetic samples are then used VLMs for embeddings to construct expressive functional priors. Furthermore, the proposed method is evaluated against various baselines, and experimental results demonstrate that our method consistently improves predictive performance and yields more reliable uncertainty estimates, particularly in out-of-distribution (OOD) detection tasks and data-scarce regimes.

Function-Space Empirical Bayes Regularisation with Large Vision-Language Model Priors

TL;DR

The paper tackles uncertainty quantification for high-dimensional deep learning by coupling function-space empirical Bayes with large vision-language models. It introduces VLM-FS-EB, which generates semantically meaningful context points via VLMs and forms expressive function-space priors from frozen VLM embeddings, enabling robust posterior inference with MC dropout. Empirical results across multiple datasets show improved predictive performance and, crucially, superior out-of-distribution detection, particularly in data-scarce regimes. This approach reduces reliance on task-specific pretraining data while leveraging foundation-model representations to achieve reliable uncertainty estimates in safety-critical settings.

Abstract

Bayesian deep learning (BDL) provides a principled framework for reliable uncertainty quantification by combining deep neural networks with Bayesian inference. A central challenge in BDL lies in the design of informative prior distributions that scale effectively to high-dimensional data. Recent functional variational inference (VI) approaches address this issue by imposing priors directly in function space; however, most existing methods rely on Gaussian process (GP) priors, whose expressiveness and generalisation capabilities become limited in high-dimensional regimes. In this work, we propose VLM-FS-EB, a novel function-space empirical Bayes regularisation framework, leveraging large vision-language models (VLMs) to generates semantically meaningful context points. These synthetic samples are then used VLMs for embeddings to construct expressive functional priors. Furthermore, the proposed method is evaluated against various baselines, and experimental results demonstrate that our method consistently improves predictive performance and yields more reliable uncertainty estimates, particularly in out-of-distribution (OOD) detection tasks and data-scarce regimes.
Paper Structure (26 sections, 22 equations, 1 figure, 12 tables)

This paper contains 26 sections, 22 equations, 1 figure, 12 tables.

Table of Contents

  1. INTRODUCTION
  2. Related Work
  3. Function-space Variational Inference Regularisation
  4. Large Vision-Language Models for Embeddings and Data Generation
  5. Synthetic Data Generation with Large Models for Uncertainty Quantification
  6. Vision-Language Models as Feature Extractors
  7. Preliminary
  8. Vision-Language Models for Synthesis and Embedding
  9. Function-Space Empirical Bayes Regularisation
  10. Proposed Method: VLM-FS-EB
  11. Context Point Sampling via Large Vision-Language Models
  12. Functional Priors with Large Vision-Language Embedding Models
  13. Scalable Posterior Inference via MC Dropout
  14. Variational approximation via MC dropout.
  15. Empirical Evaluation
  16. ...and 11 more sections

Figures (1)

  • Figure 1: Illustration of the VLM-FS-EB prior framework. Block I shows the pipeline for generating context points $\mathbf{x}_c$ via VLMs. Block II depicts how these synthetic context points are used to construct the VLM-FS-EB functional prior via a VLM for embeddings $h_{L}(\cdot)$.

Theorems & Definitions (2)

  • Remark 1
  • Remark 2