Neural Variational Dropout Processes

TL;DR

This work tackles robust meta-learning under data scarcity by modeling task-specific uncertainty through a conditional dropout posterior. It introduces Neural Variational Dropout Processes (NVDPs), which predict per-parameter dropout rates via a memory-efficient low-rank Bernoulli expert meta-model, enabling quick adaptation of a shared neural network to new tasks. A novel variational prior conditioned on the full task data regularizes the amortized VI objective, improving both uncertainty estimation and generalization. Across 1D regression, 2D image completion, and few-shot classification, NVDPs achieve strong log-likelihoods, meaningful sample variability, and competitive accuracy, demonstrating scalable, uncertainty-aware meta-learning.

Abstract

Learning to infer the conditional posterior model is a key step for robust meta-learning. This paper presents a new Bayesian meta-learning approach called Neural Variational Dropout Processes (NVDPs). NVDPs model the conditional posterior distribution based on a task-specific dropout; a low-rank product of Bernoulli experts meta-model is utilized for a memory-efficient mapping of dropout rates from a few observed contexts. It allows for a quick reconfiguration of a globally learned and shared neural network for new tasks in multi-task few-shot learning. In addition, NVDPs utilize a novel prior conditioned on the whole task data to optimize the conditional \textit{dropout} posterior in the amortized variational inference. Surprisingly, this enables the robust approximation of task-specific dropout rates that can deal with a wide range of functional ambiguities and uncertainties. We compared the proposed method with other meta-learning approaches in the few-shot learning tasks such as 1D stochastic regression, image inpainting, and classification. The results show the excellent performance of NVDPs.

Figures (11)

• Figure 1: (a) The low-rank product of Bernoulli experts meta-model of conditional dropout posterior. (b) The probabilistic graphical model of Neural Variational Dropout Processes (NVDPs) with the variational prior. Where $\{x_i^{t}, y_i^{t}\}_{i=1}^{N}$ is the N i.i.d samples from the $t$-th training dataset $\mathcal{D}^{t}$ among $T$ tasks. The context set $\mathcal{D}_C^{t}=\{x_i^{t}, y_i^{t}\}_{i=1}^{S}$ is a small subset of the $t$-th training dataset.
• Figure 2: The 1D few-shot regression results of the models on GP dataset in fixed variance and learned variance settings. The black (dash-line) represent the true unknown task function. Black dots are a few context points ($S=5$) given to the posteriors. The blue lines (and light blue area in learned variance settings) are mean values (and variance) predicted from the sampled NNs.
• Figure 3: Active learning performance on regression after up to 19 selected data points. NVDPs can use its uncertainty estimation to quickly improve LLs, while other models are learning slowly.
• Figure 4: The results from the 2D image completion tasks on CelebA, MNIST, and Omniglot dataset. Given the observed context points (10, 30, 100, half, and full pixels), the mean values of two independently sampled functions from the models (i.e. NP, NP+CNP and NVDP (ours)) are presented.
• Figure 5: The additional 1D few-shot regression results of the models on GP dataset in fixed variance settings. The black dotted lines represent the true unknown task functions. Black dots are a few context points ($S=5$) given to the posteriors. The blue lines are mean values predicted from the sampled NNs.