Learning from Aggregate responses: Instance Level versus Bag Level Loss Functions

TL;DR

This work studies learning from aggregate responses to preserve privacy, focusing on bag-level and instance-level losses and their bias-variance implications. It reveals that the instance-level loss equals a regularized bag-level loss and introduces a tunable interpolating estimator with parameter $\rho$ to balance bias and variance. In a precise linear-model theory under the proportional regime, the authors derive asymptotic bias, variance, and risk for both losses, establish a SNR-based threshold for when each method dominates, and extend the framework to differential privacy with optimal bag-size guidance. Empirical results on synthetic data and real datasets validate the theory and demonstrate the practical value of the interpolating estimator for privacy-preserving learning from aggregates.

Abstract

Due to the rise of privacy concerns, in many practical applications the training data is aggregated before being shared with the learner, in order to protect privacy of users' sensitive responses. In an aggregate learning framework, the dataset is grouped into bags of samples, where each bag is available only with an aggregate response, providing a summary of individuals' responses in that bag. In this paper, we study two natural loss functions for learning from aggregate responses: bag-level loss and the instance-level loss. In the former, the model is learnt by minimizing a loss between aggregate responses and aggregate model predictions, while in the latter the model aims to fit individual predictions to the aggregate responses. In this work, we show that the instance-level loss can be perceived as a regularized form of the bag-level loss. This observation lets us compare the two approaches with respect to bias and variance of the resulting estimators, and introduce a novel interpolating estimator which combines the two approaches. For linear regression tasks, we provide a precise characterization of the risk of the interpolating estimator in an asymptotic regime where the size of the training set grows in proportion to the features dimension. Our analysis allows us to theoretically understand the effect of different factors, such as bag size on the model prediction risk. In addition, we propose a mechanism for differentially private learning from aggregate responses and derive the optimal bag size in terms of prediction risk-privacy trade-off. We also carry out thorough experiments to corroborate our theory and show the efficacy of the interpolating estimator.

Figures (4)

• Figure 1: Effect of SNR, bag size ($k$) and overparametrization $(1/\psi = d/n)$ on the bias, variance and the risk of the model. The curves are the theoretical curves given by Theorem \ref{['thm:main']}.
• Figure 2: Label differentially private learning from aggregate data
• Figure 3: Solid curves correspond are the theoretical curves (Theorem \ref{['thm:main']}) and the dots (symbols) correspond to simulations. Here $d = 100$, and we already see a perfect match between the (asymptotic) theory and the simulations.
• Figure 4: Test loss versus the regularization parameter $\rho$ for different bag sizes, on the Boston Housing dataset. The values on the lines represent the mean of the test loss and the error bars represent the standard deviation of the test loss. The red dots denote the optimal value of $\rho$ that achieves the minimum test loss.

Theorems & Definitions (10)

• Lemma 2.1
• Lemma 2.2
• Theorem 2.5
• Corollary 3.1
• Lemma 3.2
• Definition 4.1
• Lemma 4.2
• Theorem 4.3
• Lemma B.1
• Lemma B.2