Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Gossip Algorithms for Machine Learning with Pairwise Objectives

Igor Colin, Aurélien Bellet, Stephan Clémençon, Joseph Salmon

Abstract

In the IoT era, information is more and more frequently picked up by connected smart sensors with increasing, though limited, storage, communication and computation abilities. Whether due to privacy constraints or to the structure of the distributed system, the development of statistical learning methods dedicated to data that are shared over a network is now a major issue. Gossip-based algorithms have been developed for the purpose of solving a wide variety of statistical learning tasks, ranging from data aggregation over sensor networks to decentralized multi-agent optimization. Whereas the vast majority of contributions consider situations where the function to be estimated or optimized is a basic average of individual observations, it is the goal of this article to investigate the case where the latter is of pairwise nature, taking the form of a U -statistic of degree two. Motivated by various problems such as similarity learning, ranking or clustering for instance, we revisit gossip algorithms specifically designed for pairwise objective functions and provide a comprehensive theoretical framework for their convergence. This analysis fills a gap in the literature by establishing conditions under which these methods succeed, and by identifying the graph properties that critically affect their efficiency. In particular, a refined analysis of the convergence upper and lower bounds is performed.

On Gossip Algorithms for Machine Learning with Pairwise Objectives

Abstract

In the IoT era, information is more and more frequently picked up by connected smart sensors with increasing, though limited, storage, communication and computation abilities. Whether due to privacy constraints or to the structure of the distributed system, the development of statistical learning methods dedicated to data that are shared over a network is now a major issue. Gossip-based algorithms have been developed for the purpose of solving a wide variety of statistical learning tasks, ranging from data aggregation over sensor networks to decentralized multi-agent optimization. Whereas the vast majority of contributions consider situations where the function to be estimated or optimized is a basic average of individual observations, it is the goal of this article to investigate the case where the latter is of pairwise nature, taking the form of a U -statistic of degree two. Motivated by various problems such as similarity learning, ranking or clustering for instance, we revisit gossip algorithms specifically designed for pairwise objective functions and provide a comprehensive theoretical framework for their convergence. This analysis fills a gap in the literature by establishing conditions under which these methods succeed, and by identifying the graph properties that critically affect their efficiency. In particular, a refined analysis of the convergence upper and lower bounds is performed.
Paper Structure (46 sections, 28 theorems, 202 equations, 1 figure, 7 algorithms)

This paper contains 46 sections, 28 theorems, 202 equations, 1 figure, 7 algorithms.

Table of Contents

  1. Introduction
  2. Background and Preliminaries
  3. Decentralized Setup - Motivation and Framework
  4. Formal setup.
  5. Gossip algorithms.
  6. Pairwise Averages and U-statistics: Definition and Examples
  7. Decentralized Estimation of a Pairwise Functional
  8. GoSta Algorithm
  9. Non-asymptotic Estimation Bounds
  10. Decentralized Optimization of a Pairwise Objective Function
  11. Decentralized Pairwise Function Optimization
  12. Average iterate and gradient bias.
  13. Pairwise Ergodic Dual Averaging - Gradient Bias Convergence Analysis
  14. A Lower Bound Result for Pairwise Decentralized Optimization
  15. Numerical Experiments
  16. ...and 31 more sections

Key Result

Proposition 1

Let $\mathcal{G} = ([n], \mathcal{E})$ be a connected and non-bipartite graph, $(\mathbf{x}_1, \ldots, \mathbf{x}_n)$ a sample of $n\geq 2$ points in $\mathcal{X}\subset\mathbb{R}^{d}$ and $(\mathbf{z}(t))_{t\geq 1}$ the sequence of estimates generated by Algorithm alg:gosta-sync. For all $k \in [n] where $\lambda_{n - 1}$ is the second smallest eigenvalue of the graph Laplacian $\mathbf{L}$ and

Figures (1)

  • Figure 1: Logistic AUC. Solid lines are averages and filled area are standard deviations.

Theorems & Definitions (51)

  • Remark
  • Proposition 1: Convergence in expectation
  • Theorem 1: Expected deviation
  • Proposition 1
  • Theorem 2: Pairwise ergodic dual averaging
  • proof : Sketch of proof
  • Corollary 1: Corrected and refined rate for $\gamma(t)=a/\sqrt{t}$
  • Remark
  • Theorem 3: Lower bound
  • Proposition 2
  • ...and 41 more