Theoretical Proportion Label Perturbation for Learning from Label Proportions in Large Bags

TL;DR

Experimental results demonstrate that the proportion label perturbation and loss weighting achieve classification accuracy comparable to that obtained without sampling, and this study aims to develop an LLP method capable of learning from bags with large sizes.

Abstract

Learning from label proportions (LLP) is a kind of weakly supervised learning that trains an instance-level classifier from label proportions of bags, which consist of sets of instances without using instance labels. A challenge in LLP arises when the number of instances in a bag (bag size) is numerous, making the traditional LLP methods difficult due to GPU memory limitations. This study aims to develop an LLP method capable of learning from bags with large sizes. In our method, smaller bags (mini-bags) are generated by sampling instances from large-sized bags (original bags), and these mini-bags are used in place of the original bags. However, the proportion of a mini-bag is unknown and differs from that of the original bag, leading to overfitting. To address this issue, we propose a perturbation method for the proportion labels of sampled mini-bags to mitigate overfitting to noisy label proportions. This perturbation is added based on the multivariate hypergeometric distribution, which is statistically modeled. Additionally, loss weighting is implemented to reduce the negative impact of proportions sampled from the tail of the distribution. Experimental results demonstrate that the proportion label perturbation and loss weighting achieve classification accuracy comparable to that obtained without sampling. Our codes are available at https://github.com/stainlessnight/LLP-LargeBags.

Figures (7)

• Figure 1: Learning from label proportions (LLP). An instance-level classifier is trained using bag-level labels (label proportion in a bag), where a bag consists of a set of instances and the labels of individual instances are unknown.
• Figure 2: (Left): Mean absolute error (MAE) of label proportions between the mini-bag's ground truth and the original bag when changing the sample size. The error bar shows the standard deviation. (Right): Classification accuracy of each sample size. Red and blue lines indicate the results on CIFAR-10 and SVHN, respectively. Since the MAEs are mostly the same on both datasets, SVHN is omitted in the left figure.
• Figure 3: Overview of the perturbation for the proportion of mini-bags. Random sampling generates a mini-bag from the original bag in each iteration. The supervision for the proportion of the mini-bags is also randomly determined along with the multivariate hypergeometric distribution. This perturbation can mitigate overfitting to the noisy proportion.
• Figure 4: Direction of gradients at each iteration. (a) A case when using original bag label proportions. The estimated proportion converged to the noisy proportion (overfitting). (b) A case when using label proportions sampled from the multivariate hypergeometric distribution. It does not converge to the noisy proportion due to perturbation.
• Figure 5: Training accuracy (blue) and loss (red) of the baseline method (left figure) and our proposed method (right figure).