Efficiency for Free: Ideal Data Are Transportable Representations

TL;DR

This work proposes the Representation Learning Accelerator (\algopt), which promotes the formation and utilization of efficient data, thereby accelerating representation learning.

Abstract

Data, the seminal opportunity and challenge in modern machine learning, currently constrains the scalability of representation learning and impedes the pace of model evolution. In this work, we investigate the efficiency properties of data from both optimization and generalization perspectives. Our theoretical and empirical analysis reveals an unexpected finding: for a given task, utilizing a publicly available, task- and architecture-agnostic model (referred to as the `prior model' in this paper) can effectively produce efficient data. Building on this insight, we propose the Representation Learning Accelerator (\algopt), which promotes the formation and utilization of efficient data, thereby accelerating representation learning. Utilizing a ResNet-18 pre-trained on CIFAR-10 as a prior model to inform ResNet-50 training on ImageNet-1K reduces computational costs by 50% while maintaining the same accuracy as the model trained with the original BYOL, which requires 100% cost. Our code is available at: \url{https://github.com/LINs-lab/ReLA}.

Figures (7)

• Figure 1: Framework and Intuition of ReLA: (1) Framework: ReLA serves as both a data optimizer and an auxiliary accelerator. Initially, it operates as a data optimizer by leveraging an dataset and a pre-trained model (e.g., one sourced from online repositories) to generate an efficient dataset. Subsequently, ReLA functions as an auxiliary accelerator, enhancing existing (self-)supervised learning algorithms through the effective utilization of the efficient dataset, thereby promoting efficient representation learning. (2) Intuition: The central concept of ReLA is to create an efficient-data-driven shortcut pathway within the learning process, enabling the initial model $\boldsymbol{\phi}$ to rapidly converge towards a 'proximal representation $\boldsymbol{\psi}$' of the target model $\boldsymbol{\phi}^\star$ during the early stages of training. This approach significantly accelerates the overall learning process.
• Figure 2: Investigating modified samples with varied $\Sigma$ values. Following li2018visualizing, Figure \ref{['fig:original_sample']} visualizes the validation loss landscape within a two-dimensional parameter space, along with three training trajectories corresponding to different $\Sigma$ settings. Figure \ref{['fig:diff_sigma']} illustrates the performance of models trained using samples with varied $\Sigma$. The optimal case in our task, utilizing samples with $\Sigma=0.1$ (which achieves the lowest validation loss in Figure \ref{['fig:diff_sigma']}), is visualized in Figure \ref{['fig:optimal_sample']}, where the color bar represents the values of targets $y$.
• Figure 3: Investigating modified targets with varied $\rho$ values. We present a visualization of the validation loss landscape in Figure \ref{['fig:label_landscape']}, including three training trajectories that correspond to different $\rho$ settings. Figure \ref{['fig:diff_rho']} illustrates the performance of models trained using targets with varying $\rho$ values. The optimal scenario for our task, which uses targets with $\rho=1.0$, is depicted in Figure \ref{['fig:optimal_label']}.
• Figure 4: Ablation study on BYOL () components and parameters. (a) We analyze the representation similarity between various source models (indicated on the x-axis) and target models (indicated on the y-axis). (b) We compare the static ReLA weight setting strategy with our adaptive strategy. Dotted lines ('- -') represent our adaptive strategy, while solid lines ('---') denote the static $\lambda$ setting strategy. Specifically, in the static weight setting (e.g., $0.4$), the first $40\%$ of the training leverages ReLA, with the remaining $60\%$ employing the original algorithm. (c) We present the computational cost, quantified as training time/steps, of our ReLA across various prior models.
• Figure 5: Comparison of training dynamics between ReLA and the original (Orig.) BYOL algorithm.

Theorems & Definitions (32)

• Definition 1: Supervised learning over data
• Definition 2: Data-efficient Learning
• Definition 3: Bimodal Gaussian mixture distribution
• Theorem 1: Convergence rate of learning on efficient samples
• Theorem 2: Convergence rate of learning on re-labeled data
• Remark 1: Ideal data properties avoid implicitly introduced gradient noise from data
• Remark 2: Imperfect mappings and inaccurate targets in real-world datasets
• Definition 4: Representation distance
• Theorem 3: Generalization bound with labeler $\boldsymbol{\psi}$
• Definition 5: Properties of ideal efficient data, including samples $S_X$ and targets $S_Y$