Routing the Lottery: Adaptive Subnetworks for Heterogeneous Data

TL;DR

Routing the Lottery reframes pruning as a data-driven modularization problem by learning multiple adaptive tickets instead of a single universal mask. The method partitions data into subsets, learns subset-specific sparse masks, and jointly retrains to preserve shared backbone structure while avoiding interference. Across CIFAR-10/100, implicit representations, and speech enhancement, RTL achieves higher accuracy/recall or perceptual metrics with far fewer parameters than independent models, while revealing a diagnostic subnetwork similarity score to predict oversparsification. This work demonstrates that aligning model sparsity with data heterogeneity yields modular, scalable, and interpretable deep models with practical efficiency gains.

Abstract

In pruning, the Lottery Ticket Hypothesis posits that large networks contain sparse subnetworks, or winning tickets, that can be trained in isolation to match the performance of their dense counterparts. However, most existing approaches assume a single universal winning ticket shared across all inputs, ignoring the inherent heterogeneity of real-world data. In this work, we propose Routing the Lottery (RTL), an adaptive pruning framework that discovers multiple specialized subnetworks, called adaptive tickets, each tailored to a class, semantic cluster, or environmental condition. Across diverse datasets and tasks, RTL consistently outperforms single- and multi-model baselines in balanced accuracy and recall, while using up to 10 times fewer parameters than independent models and exhibiting semantically aligned. Furthermore, we identify subnetwork collapse, a performance drop under aggressive pruning, and introduce a subnetwork similarity score that enables label-free diagnosis of oversparsification. Overall, our results recast pruning as a mechanism for aligning model structure with data heterogeneity, paving the way toward more modular and context-aware deep learning.

Figures (12)

• Figure 1: Adaptive pruning pipeline. First, the dataset is divided into subsets via predefined clustering. Then we extracts adaptive tickets, i.e. subnetworks, optimized to specific data cluster, and finally we performed network joint retraining.
• Figure 2: Mask collapse analysis on CIFAR-10 and CIFAR-100. Each subplot corresponds to one class and shows balanced accuracy (solid line) and mask similarity to other subnetworks (dashed line). Plots A-J corresponds to CIFAR-10 network and plots K-S to CIFAR-100.
• Figure 3: Semantic and structural correlation analysis. (A) Spearman's rank-order correlation between semantic similarity and mask similarity versus pruning ratio across shallow, middle, and deep layers. (B) Correlation across depth for early, middle, and late training stages. (C) Correlation versus pruning ratio for four representative classes: airplane, cat, deer, and truck. (D) Correlation across depth for the same four classes. (E) WordNet path similarity (top) and RTL mask similarity matrices for shallow (middle) and deep (bottom) layers.
• Figure 4: Data samples from the ADE20K dataset used in the INR experiments, shown together with their corresponding preprocessed semantic segmentation masks.
• Figure 5: Per-image relationship between reconstruction quality and pruning mask similarity in the INR experiment. For each ADE20K image (A–J), PSNR (orange, left axis) and average mask similarity measured by the Jaccard index (blue dashed, right axis) are shown as a function of sparsity.