Transferable and Principled Efficiency for Open-Vocabulary Segmentation

TL;DR

The core strategy is to make the efficiency principled and thus seamlessly transferable from one OVS framework to others without further customization, by utilizing smaller models that incur lower training costs.

Abstract

Recent success of pre-trained foundation vision-language models makes Open-Vocabulary Segmentation (OVS) possible. Despite the promising performance, this approach introduces heavy computational overheads for two challenges: 1) large model sizes of the backbone; 2) expensive costs during the fine-tuning. These challenges hinder this OVS strategy from being widely applicable and affordable in real-world scenarios. Although traditional methods such as model compression and efficient fine-tuning can address these challenges, they often rely on heuristics. This means that their solutions cannot be easily transferred and necessitate re-training on different models, which comes at a cost. In the context of efficient OVS, we target achieving performance that is comparable to or even better than prior OVS works based on large vision-language foundation models, by utilizing smaller models that incur lower training costs. The core strategy is to make our efficiency principled and thus seamlessly transferable from one OVS framework to others without further customization. Comprehensive experiments on diverse OVS benchmarks demonstrate our superior trade-off between segmentation accuracy and computation costs over previous works. Our code is available on https://github.com/Xujxyang/OpenTrans

Figures (9)

• Figure 1: Comparison with popular open-vocabulary segmentation works on Pascal-Context mottaghi2014role (with Resnet50 as backbone). The red points represent our works, the orange points stand for traditional two-stage open-vocabulary segmentation ding2022decouplingxu2022simple, the blue point stands for single-stage open-vocabulary segmentation han2023global, the green denotes a novel dual-channel prediction model with frozen backbone yu2024convolutions, the purple represent traditional convolutional segmentation networks chen2017rethinking.
• Figure 2: Overview of OpenTrans(Ours). We introduce principled and transferable efficiency in two folds. Step1: we prune the heavy CLIP image encoder without semantic awareness, this turns the backbone into a semantic-agnostic transferable subnetwork. This allows us to seamlessly transfer subnetwork to other OVS Frameworks, such as Deeplabv3 chen2017rethinking, Mask2former cheng2022masked and FC-CLIP yu2024convolutions. Step2: during fine-tuning, we further explore and prioritize principled efficiency by introducing layer-wise heavy-tail spectrum analysis. This method involves selectively updating layers with light-tail spectra in their pretrained weights, while keeping layers with heavy-tail spectra frozen.
• Figure 3: We transfer our semantic-agnostic sparse masks to different OVS pipelines. Left: Traditional CNN-based frameworks: these frameworks commonly employ simple upsampling techniques. Middle: Mask2Former-based frameworks, the approach utilizes a decoupled mask and class segmentation head. Right: OVS with extra classifiers: some frameworks propose to leverage a frozen CLIP classifier to further utilize the pretrained visual knowledge (with the blue box indicating frozen classifier layers of the model).
• Figure 4: We partially fine-tune layers of our pretrained backbone by analyzing heavy-tail spectrum of pretrained weights. We first fine-tune the whole model for $10^4$ iterations. Then, we compute $\alpha$ values for all layers of the model. During fine-tuning, we freeze layers with small $\alpha$ values (indicating good pretrained quality) and only fine-tune layers with large $\alpha$ (bad pretrained quality).
• Figure 5: Visualizations of examples on the PC-59 validation set by our models (trained on COCO panoptic training set, zero-shot evaluated on the PC-59 validation set).