LLP-Bench: A Large Scale Tabular Benchmark for Learning from Label Proportions
Anand Brahmbhatt, Mohith Pokala, Rishi Saket, Aravindan Raghuveer
TL;DR
LLP-Bench addresses the lack of a large-scale tabular LLP benchmark by constructing a diverse suite of about $56$ LLP datasets derived from the Criteo CTR and SSCL corpora, including both feature bags and random bags; it introduces four hardness metrics to quantify dataset difficulty and systematically evaluates a broad set of LLP baselines. The approach combines careful bag creation and filtering with a consistent evaluation protocol using a simple two-layer perceptron, providing insights into how bag structure and label variation affect learning from label proportions. Key findings show that SIM-LLP and DLLP-based losses often outperform alternatives on feature bags, while GenBags can excel on random bags, especially at larger bag sizes; the results also reveal meaningful correlations between hardness metrics and baseline performance. Overall, LLP-Bench enables rigorous, scalable, and interpretable benchmarking of LLP methods, with practical implications for privacy-preserving learning on large tabular data and a foundation for future method development.
Abstract
In the task of Learning from Label Proportions (LLP), a model is trained on groups (a.k.a bags) of instances and their corresponding label proportions to predict labels for individual instances. LLP has been applied pre-dominantly on two types of datasets - image and tabular. In image LLP, bags of fixed size are created by randomly sampling instances from an underlying dataset. Bags created via this methodology are called random bags. Experimentation on Image LLP has been mostly on random bags on CIFAR-* and MNIST datasets. Despite being a very crucial task in privacy sensitive applications, tabular LLP does not yet have a open, large scale LLP benchmark. One of the unique properties of tabular LLP is the ability to create feature bags where all the instances in a bag have the same value for a given feature. It has been shown in prior research that feature bags are very common in practical, real world applications [Chen et. al '23, Saket et. al. '22]. In this paper, we address the lack of a open, large scale tabular benchmark. First we propose LLP-Bench, a suite of 70 LLP datasets (62 feature bag and 8 random bag datasets) created from the Criteo CTR prediction and the Criteo Sponsored Search Conversion Logs datasets, the former a classification and the latter a regression dataset. These LLP datasets represent diverse ways in which bags can be constructed from underlying tabular data. To the best of our knowledge, LLP-Bench is the first large scale tabular LLP benchmark with an extensive diversity in constituent datasets. Second, we propose four metrics that characterize and quantify the hardness of a LLP dataset. Using these four metrics we present deep analysis of the 62 feature bag datasets in LLP-Bench. Finally we present the performance of 9 SOTA and popular tabular LLP techniques on all the 62 datasets.