Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Networked Information Aggregation via Machine Learning

Michael Kearns, Aaron Roth, Emily Ryu

TL;DR

$We study distributed regression with agents placed on a directed acyclic graph (DAG) where each agent observes only a subset of features and learns sequentially by incorporating its parents\' predictions as inputs. The central notion is information aggregation: achieving an overall predictor competitive with the best linear (or more general) predictor trained on the union of all features, despite each agent seeing only part of the feature set. A key insight is that the depth of the DAG—the length of the longest path—controls the ability to aggregate information; the paper develops upper and lower bounds showing depth governs excess error for both linear and general hypothesis classes, and introduces the Greedy Orthogonal Regression framework to extend guarantees to non-linear models. Complementing theory, experiments on real datasets demonstrate depth- and topology-related improvements and provide empirical guidance on how network structure affects learning performance.}

Abstract

We study a distributed learning problem in which learning agents are embedded in a directed acyclic graph (DAG). There is a fixed and arbitrary distribution over feature/label pairs, and each agent or vertex in the graph is able to directly observe only a subset of the features -- potentially a different subset for every agent. The agents learn sequentially in some order consistent with a topological sort of the DAG, committing to a model mapping observations to predictions of the real-valued label. Each agent observes the predictions of their parents in the DAG, and trains their model using both the features of the instance that they directly observe, and the predictions of their parents as additional features. We ask when this process is sufficient to achieve \emph{information aggregation}, in the sense that some agent in the DAG is able to learn a model whose error is competitive with the best model that could have been learned (in some hypothesis class) with direct access to \emph{all} features, despite the fact that no single agent in the network has such access. We give upper and lower bounds for this problem for both linear and general hypothesis classes. Our results identify the \emph{depth} of the DAG as the key parameter: information aggregation can occur over sufficiently long paths in the DAG, assuming that all of the relevant features are well represented along the path, and there are distributions over which information aggregation cannot occur even in the linear case, and even in arbitrarily large DAGs that do not have sufficient depth (such as a hub-and-spokes topology in which the spoke vertices collectively see all the features). We complement our theoretical results with a comprehensive set of experiments.

Networked Information Aggregation via Machine Learning

TL;DR

$We study distributed regression with agents placed on a directed acyclic graph (DAG) where each agent observes only a subset of features and learns sequentially by incorporating its parents\' predictions as inputs. The central notion is information aggregation: achieving an overall predictor competitive with the best linear (or more general) predictor trained on the union of all features, despite each agent seeing only part of the feature set. A key insight is that the depth of the DAG—the length of the longest path—controls the ability to aggregate information; the paper develops upper and lower bounds showing depth governs excess error for both linear and general hypothesis classes, and introduces the Greedy Orthogonal Regression framework to extend guarantees to non-linear models. Complementing theory, experiments on real datasets demonstrate depth- and topology-related improvements and provide empirical guidance on how network structure affects learning performance.}

Abstract

We study a distributed learning problem in which learning agents are embedded in a directed acyclic graph (DAG). There is a fixed and arbitrary distribution over feature/label pairs, and each agent or vertex in the graph is able to directly observe only a subset of the features -- potentially a different subset for every agent. The agents learn sequentially in some order consistent with a topological sort of the DAG, committing to a model mapping observations to predictions of the real-valued label. Each agent observes the predictions of their parents in the DAG, and trains their model using both the features of the instance that they directly observe, and the predictions of their parents as additional features. We ask when this process is sufficient to achieve \emph{information aggregation}, in the sense that some agent in the DAG is able to learn a model whose error is competitive with the best model that could have been learned (in some hypothesis class) with direct access to \emph{all} features, despite the fact that no single agent in the network has such access. We give upper and lower bounds for this problem for both linear and general hypothesis classes. Our results identify the \emph{depth} of the DAG as the key parameter: information aggregation can occur over sufficiently long paths in the DAG, assuming that all of the relevant features are well represented along the path, and there are distributions over which information aggregation cannot occur even in the linear case, and even in arbitrarily large DAGs that do not have sufficient depth (such as a hub-and-spokes topology in which the spoke vertices collectively see all the features). We complement our theoretical results with a comprehensive set of experiments.

Paper Structure

This paper contains 44 sections, 43 theorems, 166 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Our Results and Techniques
  3. Intuition for Non-Triviality.
  4. Upper Bound in the Linear Case.
  5. Beyond Linear Models
  6. Lower Bounds
  7. Experiments
  8. Additional Related Work
  9. Opinion Dynamics and Consensus Models
  10. Bayesian Learning in Graphs
  11. Information Aggregation Using Tractable Calibration Conditions
  12. Preliminaries
  13. Linear Learners
  14. Structural Properties of Squared Error Regression
  15. Analysis of Learners Along a Path
  16. ...and 29 more sections

Key Result

Lemma 3.1

If $f(x)$ is the least squares estimator for a target $y$ using features $x \in \mathbb{R}^k$ (i.e., $f(x) = w^{*T}x$ where $w^* = \arg\min_w \mathbb{E}[(y - w^T x)^2]$), then the residuals of $f(x)$ are orthogonal to each of its features. In other words for each index $j \in [k]$: This implies that $f$ is multiaccurate with respect to the collection of all linear functions $g(x) = a^T x$ for any

Figures (4)

  • Figure 1: Simulation results for lower bound construction. See text for discussion.
  • Figure 2: Experimental results for linear models. Left column corresponds to wine quality dataset, right column to appliance energy usage dataset. See text for details.
  • Figure 3: Sample top-down tree of 50 learners.
  • Figure 4: Experimental results for neural networks. Left column corresponds to wine quality dataset, right column to appliance energy usage dataset. See text for details.

Theorems & Definitions (95)

  • Definition 2.1: Mean Squared Error
  • Definition 2.2: The $L_2$ norm of a random variable
  • Definition 2.3: Multiaccuracy
  • Definition 2.4
  • Lemma 3.1
  • proof
  • Corollary 3.2
  • proof
  • Lemma 3.3: MSE Decomposition
  • proof
  • ...and 85 more