DANCE: DAta-Network Co-optimization for Efficient Segmentation Model Training and Inference

TL;DR

DANCE introduces a data-network co-optimization framework for semantic segmentation, combining spatial-complexity–guided data slimming with progressive network slimming to reduce training and inference costs while maintaining or improving accuracy. By mapping a per-image spatial complexity score to downsampling, dropping, and loss weighting, and by progressively pruning multi-scale modules, DANCE achieves substantial energy savings across diverse models and datasets. Extensive experiments across Cityscapes, CamVid, and BDD with four segmentation backbones demonstrate an all-win trade-off: lower computation and energy with equal or better mIoU, and robust gains across edge hardware. The work also shows that co-optimization shapes pruning patterns and that the proposed complexity-indicator outperforms alternatives, offering a practical pathway toward efficient on-device segmentation and continual learning.

Abstract

Semantic segmentation for scene understanding is nowadays widely demanded, raising significant challenges for the algorithm efficiency, especially its applications on resource-limited platforms. Current segmentation models are trained and evaluated on massive high-resolution scene images ("data level") and suffer from the expensive computation arising from the required multi-scale aggregation("network level"). In both folds, the computational and energy costs in training and inference are notable due to the often desired large input resolutions and heavy computational burden of segmentation models. To this end, we propose DANCE, general automated DAta-Network Co-optimization for Efficient segmentation model training and inference. Distinct from existing efficient segmentation approaches that focus merely on light-weight network design, DANCE distinguishes itself as an automated simultaneous data-network co-optimization via both input data manipulation and network architecture slimming. Specifically, DANCE integrates automated data slimming which adaptively downsamples/drops input images and controls their corresponding contribution to the training loss guided by the images' spatial complexity. Such a downsampling operation, in addition to slimming down the cost associated with the input size directly, also shrinks the dynamic range of input object and context scales, therefore motivating us to also adaptively slim the network to match the downsampled data. Extensive experiments and ablating studies (on four SOTA segmentation models with three popular segmentation datasets under two training settings) demonstrate that DANCE can achieve "all-win" towards efficient segmentation(reduced training cost, less expensive inference, and better mean Intersection-over-Union (mIoU)).

Figures (6)

• Figure 1: The achieved mIoU vs. the required energy cost (Left: the total training energy cost; Right: the averaged inference energy cost per image) on the Cityscapes cordts2016cityscapes test dataset. For the three segmentation models evaluated, DANCE achieves "all-win": reduced training cost, less expensive inference, and improved mIoU.
• Figure 2: An overview of the data-network co-optimization pipeline in the proposed DANCE framework.
• Figure 3: The image complexity distribution of the training sets in the three considered urban scene understanding datasets.
• Figure 4: The percentage of pruned channels for different kinds of layers under various pruning ratios on a DeepLabv3+ model (with a ResNet50 backbone and an output stride of 16) with the Cityscapes dataset.
• Figure 5: Ablation studies of data-network co-optimization in DANCE on the DeepLabv3+ model (with a ResNet50 backbone and an output stride of 16) and Cityscapes validation dataset, where FLOPs and energy are normalized to those of the baseline.