Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MYCROFT: Towards Effective and Efficient External Data Augmentation

Zain Sarwar, Van Tran, Arjun Nitin Bhagoji, Nick Feamster, Ben Y. Zhao, Supriyo Chakraborty

TL;DR

Mycroft is a data-efficient method that enables model trainers to evaluate the relative utility of different data sources while working with a constrained data-sharing budget and can pave the way for democratized training of high performance ML models.

Abstract

Machine learning (ML) models often require large amounts of data to perform well. When the available data is limited, model trainers may need to acquire more data from external sources. Often, useful data is held by private entities who are hesitant to share their data due to propriety and privacy concerns. This makes it challenging and expensive for model trainers to acquire the data they need to improve model performance. To address this challenge, we propose Mycroft, a data-efficient method that enables model trainers to evaluate the relative utility of different data sources while working with a constrained data-sharing budget. By leveraging feature space distances and gradient matching, Mycroft identifies small but informative data subsets from each owner, allowing model trainers to maximize performance with minimal data exposure. Experimental results across four tasks in two domains show that Mycroft converges rapidly to the performance of the full-information baseline, where all data is shared. Moreover, Mycroft is robust to noise and can effectively rank data owners by utility. Mycroft can pave the way for democratized training of high performance ML models.

MYCROFT: Towards Effective and Efficient External Data Augmentation

TL;DR

Mycroft is a data-efficient method that enables model trainers to evaluate the relative utility of different data sources while working with a constrained data-sharing budget and can pave the way for democratized training of high performance ML models.

Abstract

Machine learning (ML) models often require large amounts of data to perform well. When the available data is limited, model trainers may need to acquire more data from external sources. Often, useful data is held by private entities who are hesitant to share their data due to propriety and privacy concerns. This makes it challenging and expensive for model trainers to acquire the data they need to improve model performance. To address this challenge, we propose Mycroft, a data-efficient method that enables model trainers to evaluate the relative utility of different data sources while working with a constrained data-sharing budget. By leveraging feature space distances and gradient matching, Mycroft identifies small but informative data subsets from each owner, allowing model trainers to maximize performance with minimal data exposure. Experimental results across four tasks in two domains show that Mycroft converges rapidly to the performance of the full-information baseline, where all data is shared. Moreover, Mycroft is robust to noise and can effectively rank data owners by utility. Mycroft can pave the way for democratized training of high performance ML models.

Paper Structure

This paper contains 30 sections, 1 theorem, 7 equations, 11 figures, 5 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Problem Setup and Formulation
  3. Challenges of external data augmentation
  4. Problem Formulation and Notation
  5. Mycroft: Identifying and sharing useful data
  6. Functional Similarity via Loss Gradient Matching
  7. Feature Similarity
  8. FuncFeat : Combining Gradient and Feature Similarity
  9. Experiments
  10. Experimental Setup
  11. Results
  12. Ablation studies
  13. Case Studies
  14. Related Work
  15. Discussion
  16. ...and 15 more sections

Key Result

Theorem 3.1

If the loss function $L(\cdot)$ is bounded above by $L_{\text{max}}$ and $\, \forall j, \, \| \nabla_{\theta_i} L (z_j) \| \leq \nabla_{\text{max}}$, then $f_{\lambda_1,\lambda_2}(\mathbf{w}) = L_{\text{max}}-e'_{\lambda}(\mathbf{w})$ is weakly submodular with parameter $\gamma' \geq \frac{\lambda_{

Figures (11)

  • Figure 1: Performance of Mycroft compared to random-sampling and full-information on the tabular dataset.
  • Figure 2: Accuracy of $M_{\text{MT}}'$ when trained on $D^{\text{useful}}_{i}\xspace$ retrieved using Mycroft and random-sampling under the scenario where approximately 70% of the data or labels are corrupted.
  • Figure 3: Framework for MT to evaluate the utility of DO's data.
  • Figure 4: Top-k retrieved $D^{\text{useful}}_{i}\xspace$ samples using Unicom for $D^{\text{hard}}\xspace$ from the Dogs & Wolves dataset.
  • Figure 5: Subsets in the Dogs & Wolves dataset. The first column shows Dogs & Wolves - Spurious where the dogs are on a grass background and the wolves are on snow. The second column shows Dogs & Wolves - Natural where the dogs are on snow and the wolves are on grass.
  • ...and 6 more figures

Theorems & Definitions (3)

  • Definition 2.1: Task for each DO
  • Theorem 3.1
  • proof : Proof of Theorem \ref{['thm: submodular']}