Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerating Deep Learning with Fixed Time Budget

Muhammad Asif Khan, Ridha Hamila, Hamid Menouar

TL;DR

The results consistently show clear gains achieved by the proposed method in improving the learning performance of various state-of-the-art deep learning models in both regression and classification tasks.

Abstract

The success of modern deep learning is attributed to two key elements: huge amounts of training data and large model sizes. Where a vast amount of data allows the model to learn more features, the large model architecture boosts the learning capability of the model. However, both these factors result in prolonged training time. In some practical applications such as edge-based learning and federated learning, limited-time budgets necessitate more efficient training methods. This paper proposes an effective technique for training arbitrary deep learning models within fixed time constraints utilizing sample importance and dynamic ranking. The proposed method is extensively evaluated in both classification and regression tasks in computer vision. The results consistently show clear gains achieved by the proposed method in improving the learning performance of various state-of-the-art deep learning models in both regression and classification tasks.

Accelerating Deep Learning with Fixed Time Budget

TL;DR

The results consistently show clear gains achieved by the proposed method in improving the learning performance of various state-of-the-art deep learning models in both regression and classification tasks.

Abstract

The success of modern deep learning is attributed to two key elements: huge amounts of training data and large model sizes. Where a vast amount of data allows the model to learn more features, the large model architecture boosts the learning capability of the model. However, both these factors result in prolonged training time. In some practical applications such as edge-based learning and federated learning, limited-time budgets necessitate more efficient training methods. This paper proposes an effective technique for training arbitrary deep learning models within fixed time constraints utilizing sample importance and dynamic ranking. The proposed method is extensively evaluated in both classification and regression tasks in computer vision. The results consistently show clear gains achieved by the proposed method in improving the learning performance of various state-of-the-art deep learning models in both regression and classification tasks.
Paper Structure (21 sections, 6 equations, 5 figures, 4 tables, 1 algorithm)

This paper contains 21 sections, 6 equations, 5 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Proposed Method
  4. Theoretical Foundation
  5. Balancing Sample Diversity and Importance
  6. Computational Overhead
  7. Experiments and Results
  8. Learning Tasks
  9. Datasets
  10. CIFAR10:
  11. CIFAR100:
  12. ShanghaiTech Part-B:
  13. CARPK:
  14. Density Maps Generation:
  15. Evaluation Metrics
  16. ...and 6 more sections

Figures (5)

  • Figure 1: A simple illustration of the proposed dynamic training strategy with selective curriculum and sample importance.
  • Figure 2: Per-sample loss values for the CIFAR10 dataset computed during the initial training phase, showing large variations in loss values for different baseline models.
  • Figure 3: Comparison of loss convergence in image classification using the baseline model with standard training (solid lines) on the full dataset (CIFAR10) versus selective samples using the proposed method (dashed lines).
  • Figure 4: Illustration of important features using integrated gradients method for various image classes in CIFAR10 dataset (zoom in required for better visualization of features).
  • Figure 5: Visual qualitative analysis and comparison of the proposed method versus baseline models over ShanghaiTech Part B.