Epoch-evolving Gaussian Process Guided Learning

TL;DR

This paper proposes a novel learning scheme called epoch-evolving Gaussian Process Guided Learning (GPGL), which aims at characterizing the correlation information between the batch-level distribution and the global data distribution and provides a more efficient optimization through updating the model parameters with a triangle consistency loss.

Abstract

In this paper, we propose a novel learning scheme called epoch-evolving Gaussian Process Guided Learning (GPGL), which aims at characterizing the correlation information between the batch-level distribution and the global data distribution. Such correlation information is encoded as context labels and needs renewal every epoch. With the guidance of the context label and ground truth label, GPGL scheme provides a more efficient optimization through updating the model parameters with a triangle consistency loss. Furthermore, our GPGL scheme can be further generalized and naturally applied to the current deep models, outperforming the existing batch-based state-of-the-art models on mainstream datasets (CIFAR-10, CIFAR-100, and Tiny-ImageNet) remarkably.

Figures (3)

• Figure 1: Overview of Gaussian Process Guided Learning (GPGL) scheme. The GP model leverages the features and labels of the anchor set to construct the global data distribution approximately. The architecture of the feature extractor is the same as that of the deep model in batch-level learning process. However, the parameters of feature extractor in GP model is updated at the epoch-level. After constructing the GP model, it would make a context label $y^*_b$ for each sample, and guide the deep model learning by the triangle loss function.
• Figure 2: (a) As shown in red dashed lines, the loss functions $\mathcal{L}_{ce}^1$ and $\mathcal{L}_{kl}$ are used for updating all the parameters in the deep neural network model, while the $\mathcal{L}_{ce}^2$ is used for updating the parameters of feature extractor. (b) The context label of an image in dataset is estimated by the correlation with each class in the anchor set. The width of red solid lines indicates the probability value of the context label for each class. Wider lines show a stronger correlation.
• Figure 3: (a) Different combinations of three loss terms effect the validation error on CIFAR-10, which shows the effectiveness of our triangle loss function. (b) The training error of Resnet20 on CIFAR-100 over epochs. (c) The validation error of Resnet20 on CIFAR-100 over epochs.