Meta-Sparsity: Learning Optimal Sparse Structures in Multi-task Networks through Meta-learning

TL;DR

Meta-Sparsity introduces a meta-learning framework that learns sparsity-controlling hyperparameters to produce optimal channel-wise sparse structures in a multi-task backbone. By integrating MAML-style meta-training with group sparsity, the approach learns both model parameters and a learnable sparsity parameter, enabling dynamic, task-aware sparsity patterns that generalize to unseen tasks. Empirical results on NYU-v2 and CelebA show that meta-sparsity often outperforms dense and fixed-sparsity baselines while delivering compression and speed-up, albeit with some task-interference cases. The work highlights the potential for parsimonious, adaptable multi-task models and outlines future directions in hardware-aware sparsity and broader sparsity forms.

Abstract

This paper presents meta-sparsity, a framework for learning model sparsity, basically learning the parameter that controls the degree of sparsity, that allows deep neural networks (DNNs) to inherently generate optimal sparse shared structures in multi-task learning (MTL) setting. This proposed approach enables the dynamic learning of sparsity patterns across a variety of tasks, unlike traditional sparsity methods that rely heavily on manual hyperparameter tuning. Inspired by Model Agnostic Meta-Learning (MAML), the emphasis is on learning shared and optimally sparse parameters in multi-task scenarios by implementing a penalty-based, channel-wise structured sparsity during the meta-training phase. This method improves the model's efficacy by removing unnecessary parameters and enhances its ability to handle both seen and previously unseen tasks. The effectiveness of meta-sparsity is rigorously evaluated by extensive experiments on two datasets, NYU-v2 and CelebAMask-HQ, covering a broad spectrum of tasks ranging from pixel-level to image-level predictions. The results show that the proposed approach performs well across many tasks, indicating its potential as a versatile tool for creating efficient and adaptable sparse neural networks. This work, therefore, presents an approach towards learning sparsity, contributing to the efforts in the field of sparse neural networks and suggesting new directions for research towards parsimonious models.

Figures (14)

• Figure 1: This figure illustrates (a-d) a few common approaches to achieving sparse models and (e) the proposed approach of meta-sparsification.
• Figure 2: A schematic of the multi-task architecture used in this work (inspired by paper3).
• Figure 3: For NYU dataset, task-wise performance comparison of single-task and multi-task no sparse ($\lambda = 0$, in blue), fixed sparsity single and multi-task (for $\lambda = 1\times10^{-4}$ in yellow and $\lambda = 1\times10^{-3}$ in green) and meta sparsity ($\lambda$ = learned in red i.e., meta-sparsity) experiments. The vertical axis label is annotated with an upward or downward arrow to indicate whether a higher or lower metric value is preferable. The values below each bar represent the percentage of parameter sparsity in the backbone network. The 'x3' on the initial bar for task T4 indicates that the depicted performance metric (MAE) is triple the current value represented by the bar. This notation is employed to simplify the depiction of values that exhibit significant disparities. In these plots, we have selected a narrow range for the y-axis to make it easier to compare the performances. However, this may have enhanced the visual effect of the error bars representing the performance's standard deviation. These results are also presented in tabular form in the supplementary material Table A.1.
• Figure 4: For celebA dataset, task-wise performance comparison of single-task and multi-task no sparse ($\lambda = 0$, in blue), fixed sparsity single and multi-task (for $\lambda = 1\times10^{-5}$ in lavender, $\lambda = 1\times10^{-4}$ in yellow and $\lambda = 1\times10^{-3}$ in green) and meta sparsity (in red) experiments. The vertical axis label is annotated with an upward or downward arrow to indicate whether a higher or lower metric value is preferable. These results are also presented in tabular form in the supplementary material Table A.3.
• Figure 5: This figure presents a comparison of performance against both the compression ratio (top row) and speed-up (bottom row) across all tasks within the NYU dataset, with meta-sparsity experiments distinctly marked by red circles. It is important to note that the upward arrow ($\uparrow$) on the y-axis denotes that a higher value of the metric is preferable, and the downward arrow ($\downarrow$) represents that a lower value of the metric is preferable. The tabulated metrics are presented in Table A.4 in the supplementary material.