Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Necessity of Collaboration for Online Model Selection with Decentralized Data

Junfan Li, Zheshun Wu, Zenglin Xu, Irwin King

TL;DR

The results show that collaboration is unnecessary in the absence of computational constraints on clients and collaboration is necessary if the computational cost on each client is limited to $o(K)$, where $K$ is the number of candidate hypothesis spaces.

Abstract

We consider online model selection with decentralized data over $M$ clients, and study the necessity of collaboration among clients. Previous work proposed various federated algorithms without demonstrating their necessity,while we answer the question from a novel perspective of computational constraints. We prove lower bounds on the regret, and propose a federated algorithm and analyze the upper bound.Our results show (i) collaboration is unnecessary in the absence of computational constraints on clients; (ii) collaboration is necessary if the computational cost on each client is limited to $o(K)$, where $K$ is the number of candidate hypothesis spaces. We clarify the unnecessary nature of collaboration in previous federated algorithms for distributed online multi-kernel learning,and improve the regret bounds at a smaller computational and communication cost. Our algorithm relies on three new techniques including an improved Bernstein's inequality for martingale, a federated online mirror descent framework, and decoupling model selection and prediction, which might be of independent interest.

On the Necessity of Collaboration for Online Model Selection with Decentralized Data

TL;DR

The results show that collaboration is unnecessary in the absence of computational constraints on clients and collaboration is necessary if the computational cost on each client is limited to , where is the number of candidate hypothesis spaces.

Abstract

We consider online model selection with decentralized data over clients, and study the necessity of collaboration among clients. Previous work proposed various federated algorithms without demonstrating their necessity,while we answer the question from a novel perspective of computational constraints. We prove lower bounds on the regret, and propose a federated algorithm and analyze the upper bound.Our results show (i) collaboration is unnecessary in the absence of computational constraints on clients; (ii) collaboration is necessary if the computational cost on each client is limited to , where is the number of candidate hypothesis spaces. We clarify the unnecessary nature of collaboration in previous federated algorithms for distributed online multi-kernel learning,and improve the regret bounds at a smaller computational and communication cost. Our algorithm relies on three new techniques including an improved Bernstein's inequality for martingale, a federated online mirror descent framework, and decoupling model selection and prediction, which might be of independent interest.
Paper Structure (49 sections, 13 theorems, 150 equations, 7 tables, 6 algorithms)

This paper contains 49 sections, 13 theorems, 150 equations, 7 tables, 6 algorithms.

Table of Contents

  1. Introduction
  2. Main Results
  3. Technical Challenges
  4. Related work
  5. Preliminaries and Problem Setting
  6. Notations
  7. Online Model Selection (OMS)
  8. Online Model Selection with Decentralized Data (OMS-DecD)
  9. FOMD-No-LU
  10. Federated Algorithmic Framework
  11. Regret Bound
  12. Comparison with Previous Work
  13. OMS-DecD without Communication Constraints
  14. Decoupling Model Selection and Prediction
  15. Model Selection on Server
  16. ...and 34 more sections

Key Result

Theorem 1

Let $R=T$. Assuming that $l^{(j)}_t:\Omega\rightarrow \mathbb{R},t\in[T],j\in[M]$ is convex. Let $g^{(j)}_t=\nabla_{{\bf u}^{(j)}_t} l^{(j)}_t({\bf u}^{(j)}_t)$ and $\tilde{g}^{(j)}_t$ be an estimator of $g^{(j)}_t$. At any round $t$, let ${\bf q}_{t+1}$ and ${\bf r}_{t+1}$ be two auxiliary decision Then FOMD-No-LU guarantees that,

Theorems & Definitions (25)

  • Example 1: Online Hyper-parameters Tuning
  • Example 2: Online Kernel Selection Shen2019RandomLi2022Improved
  • Example 3: Online Pre-trained Classifier Selection Karimi2021Online
  • Theorem 1
  • Theorem 2
  • Lemma 1
  • Lemma 2
  • Theorem 3
  • Theorem 4: Lower Bounds
  • Definition 1: NCO-OMS
  • ...and 15 more