Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hardness of Learning Boolean Functions from Label Proportions

Venkatesan Guruswami, Rishi Saket

TL;DR

The paper investigates the computational hardness of learning Boolean functions from label proportions (LLP) for fundamental Boolean classes. It proves NP-hardness for LLP learning OR when the hypothesis is restricted to constant-$\ell$-clause CNFs and also rules out improving beyond $1/2+\delta$ for $\ell$-DNFs, while establishing NP-hardness for LLP learning Parities with bag size $q$. The results reuse reductions from Label-Cover and Smooth Label-Cover, employing dictatorship tests and folding into parity subspaces, and contrast with known LLP algorithms for halfspaces, thereby separating constant-clause CNFs and DNFs from halfspace-based approaches. An accompanying positive result shows a simple randomized algorithm attains $1/\,2^{q-2}$-approximation for LLP Parities, offering a near-tight bound in light of the hardness results and highlighting the nuanced landscape of LLP learnability across Boolean function classes. Overall, the work clarifies the limitations of LLP for simple Boolean function families and informs the design of LLP-based learning systems under privacy-preserving bag-level constraints.

Abstract

In recent years the framework of learning from label proportions (LLP) has been gaining importance in machine learning. In this setting, the training examples are aggregated into subsets or bags and only the average label per bag is available for learning an example-level predictor. This generalizes traditional PAC learning which is the special case of unit-sized bags. The computational learning aspects of LLP were studied in recent works (Saket, NeurIPS'21; Saket, NeurIPS'22) which showed algorithms and hardness for learning halfspaces in the LLP setting. In this work we focus on the intractability of LLP learning Boolean functions. Our first result shows that given a collection of bags of size at most $2$ which are consistent with an OR function, it is NP-hard to find a CNF of constantly many clauses which satisfies any constant-fraction of the bags. This is in contrast with the work of (Saket, NeurIPS'21) which gave a $(2/5)$-approximation for learning ORs using a halfspace. Thus, our result provides a separation between constant clause CNFs and halfspaces as hypotheses for LLP learning ORs. Next, we prove the hardness of satisfying more than $1/2 + o(1)$ fraction of such bags using a $t$-DNF (i.e. DNF where each term has $\leq t$ literals) for any constant $t$. In usual PAC learning such a hardness was known (Khot-Saket, FOCS'08) only for learning noisy ORs. We also study the learnability of parities and show that it is NP-hard to satisfy more than $(q/2^{q-1} + o(1))$-fraction of $q$-sized bags which are consistent with a parity using a parity, while a random parity based algorithm achieves a $(1/2^{q-2})$-approximation.

Hardness of Learning Boolean Functions from Label Proportions

TL;DR

The paper investigates the computational hardness of learning Boolean functions from label proportions (LLP) for fundamental Boolean classes. It proves NP-hardness for LLP learning OR when the hypothesis is restricted to constant--clause CNFs and also rules out improving beyond for -DNFs, while establishing NP-hardness for LLP learning Parities with bag size . The results reuse reductions from Label-Cover and Smooth Label-Cover, employing dictatorship tests and folding into parity subspaces, and contrast with known LLP algorithms for halfspaces, thereby separating constant-clause CNFs and DNFs from halfspace-based approaches. An accompanying positive result shows a simple randomized algorithm attains -approximation for LLP Parities, offering a near-tight bound in light of the hardness results and highlighting the nuanced landscape of LLP learnability across Boolean function classes. Overall, the work clarifies the limitations of LLP for simple Boolean function families and informs the design of LLP-based learning systems under privacy-preserving bag-level constraints.

Abstract

In recent years the framework of learning from label proportions (LLP) has been gaining importance in machine learning. In this setting, the training examples are aggregated into subsets or bags and only the average label per bag is available for learning an example-level predictor. This generalizes traditional PAC learning which is the special case of unit-sized bags. The computational learning aspects of LLP were studied in recent works (Saket, NeurIPS'21; Saket, NeurIPS'22) which showed algorithms and hardness for learning halfspaces in the LLP setting. In this work we focus on the intractability of LLP learning Boolean functions. Our first result shows that given a collection of bags of size at most which are consistent with an OR function, it is NP-hard to find a CNF of constantly many clauses which satisfies any constant-fraction of the bags. This is in contrast with the work of (Saket, NeurIPS'21) which gave a -approximation for learning ORs using a halfspace. Thus, our result provides a separation between constant clause CNFs and halfspaces as hypotheses for LLP learning ORs. Next, we prove the hardness of satisfying more than fraction of such bags using a -DNF (i.e. DNF where each term has literals) for any constant . In usual PAC learning such a hardness was known (Khot-Saket, FOCS'08) only for learning noisy ORs. We also study the learnability of parities and show that it is NP-hard to satisfy more than -fraction of -sized bags which are consistent with a parity using a parity, while a random parity based algorithm achieves a -approximation.
Paper Structure (21 sections, 16 theorems, 17 equations)

This paper contains 21 sections, 16 theorems, 17 equations.

Table of Contents

  1. Introduction
  2. Previous Related Work
  3. Overview of Our Techniques
  4. Preliminaries
  5. Problem Definitions
  6. Label Cover
  7. Smooth Label Cover
  8. Tail Bounds
  9. Hardness of LLP Learning OR using $\ell$-clause CNF
  10. Hardness Reduction
  11. NO Case
  12. Hardness of LLP Learning OR using $\ell$- DNF
  13. Hardness Reduction
  14. NO Case
  15. Hardness of LLP Learning Parities
  16. ...and 6 more sections

Key Result

Theorem 1.1

For any constants $\delta > 0, \ell \in \mathbb{Z}^+$, given a collection of bags which are of size at most $2$, and whose label proportions are consistent with some OR, it is NP-hard to compute an $\ell$-clause CNF that satisfies $\delta$-fraction of the bags.

Theorems & Definitions (23)

  • Theorem 1.1
  • Theorem 1.2
  • Theorem 1.3
  • Theorem 1.4
  • Definition 2.1
  • Theorem 2.2
  • Definition 2.3
  • Theorem 2.4
  • Lemma 2.5: Chernoff Bound
  • Theorem 3.1
  • ...and 13 more