Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TMT-VIS: Taxonomy-aware Multi-dataset Joint Training for Video Instance Segmentation

Rongkun Zheng, Lu Qi, Xi Chen, Yi Wang, Kun Wang, Yu Qiao, Hengshuang Zhao

TL;DR

A two-stage taxonomy aggregation module that first compiles taxonomy information from input videos and then aggregates these taxonomy priors into instance queries before the transformer decoder is designed, which shows significant improvement over the baseline solutions, and sets new state-of-the-art records on all benchmarks.

Abstract

Training on large-scale datasets can boost the performance of video instance segmentation while the annotated datasets for VIS are hard to scale up due to the high labor cost. What we possess are numerous isolated filed-specific datasets, thus, it is appealing to jointly train models across the aggregation of datasets to enhance data volume and diversity. However, due to the heterogeneity in category space, as mask precision increases with the data volume, simply utilizing multiple datasets will dilute the attention of models on different taxonomies. Thus, increasing the data scale and enriching taxonomy space while improving classification precision is important. In this work, we analyze that providing extra taxonomy information can help models concentrate on specific taxonomy, and propose our model named Taxonomy-aware Multi-dataset Joint Training for Video Instance Segmentation (TMT-VIS) to address this vital challenge. Specifically, we design a two-stage taxonomy aggregation module that first compiles taxonomy information from input videos and then aggregates these taxonomy priors into instance queries before the transformer decoder. We conduct extensive experimental evaluations on four popular and challenging benchmarks, including YouTube-VIS 2019, YouTube-VIS 2021, OVIS, and UVO. Our model shows significant improvement over the baseline solutions, and sets new state-of-the-art records on all benchmarks. These appealing and encouraging results demonstrate the effectiveness and generality of our approach. The code is available at https://github.com/rkzheng99/TMT-VIS .

TMT-VIS: Taxonomy-aware Multi-dataset Joint Training for Video Instance Segmentation

TL;DR

A two-stage taxonomy aggregation module that first compiles taxonomy information from input videos and then aggregates these taxonomy priors into instance queries before the transformer decoder is designed, which shows significant improvement over the baseline solutions, and sets new state-of-the-art records on all benchmarks.

Abstract

Training on large-scale datasets can boost the performance of video instance segmentation while the annotated datasets for VIS are hard to scale up due to the high labor cost. What we possess are numerous isolated filed-specific datasets, thus, it is appealing to jointly train models across the aggregation of datasets to enhance data volume and diversity. However, due to the heterogeneity in category space, as mask precision increases with the data volume, simply utilizing multiple datasets will dilute the attention of models on different taxonomies. Thus, increasing the data scale and enriching taxonomy space while improving classification precision is important. In this work, we analyze that providing extra taxonomy information can help models concentrate on specific taxonomy, and propose our model named Taxonomy-aware Multi-dataset Joint Training for Video Instance Segmentation (TMT-VIS) to address this vital challenge. Specifically, we design a two-stage taxonomy aggregation module that first compiles taxonomy information from input videos and then aggregates these taxonomy priors into instance queries before the transformer decoder. We conduct extensive experimental evaluations on four popular and challenging benchmarks, including YouTube-VIS 2019, YouTube-VIS 2021, OVIS, and UVO. Our model shows significant improvement over the baseline solutions, and sets new state-of-the-art records on all benchmarks. These appealing and encouraging results demonstrate the effectiveness and generality of our approach. The code is available at https://github.com/rkzheng99/TMT-VIS .
Paper Structure (27 sections, 4 equations, 5 figures, 12 tables)

This paper contains 27 sections, 4 equations, 5 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Overall architecture
  5. Revisiting Mask2Former.
  6. Taxonomy Compilation Module
  7. Taxonomy Injection Module
  8. Taxonomy-aware Matching Loss
  9. Implementation Details
  10. Experiments
  11. Datasets
  12. Main Results
  13. Ablation Studies
  14. Multiple dataset joint-training problem.
  15. Key component designs.
  16. ...and 12 more sections

Figures (5)

  • Figure 1: Comparison of multiple-dataset training paradigms. Previous multiple dataset training methods focus on unifying the label space, and some models adopt language embeddings to interact with queries after the decoder for the final classification. Our model (right part) leverages the taxonomic embedding to refine queries through taxonomic compilation and injection (C&I) before the decoder to enhance the performance.
  • Figure 2: Overall framework of the proposed TMT-VIS method. Taxonomy Compilation Module (TCM) adopts the CLIP text encoder and spatiotemporal Adapter to generate video-specific modulated taxonomic embeddings. The Taxonomy Injection Module (TIM) leverages the modulated embeddings to provide taxonomic guidance to visual queries in the decoder. An additional taxonomy-aware loss is added to supervise the compilation.
  • Figure 3: Visual comparison of our model with Mask2Former-VIS (abbreviated as 'M2F-VIS'). Our TMT-VIS shows better precision in segmenting small instances, e.g., the swimmers in the middle of the image, and classifying similar instances, e.g., truck and sedan are similar categories; their appearances are similar but different in sizes.
  • Figure 4: Visual comparison of our model with Mask2Former-VIS (abbreviated as 'M2F-VIS'). Our TMT-VIS shows better precision in segmenting and tracking small instances with the same taxonomy, such as the giraffes from the first two rows or the cyclists in the last two rows, and TMT-VIS shows better performance).
  • Figure 5: Visual comparison of our model with Mask2Former-VIS (abbreviated as 'M2F-VIS'). Our TMT-VIS shows better precision in segmenting and tracking instances with occlusions. In the top two rows, TMT-VIS could successfully segment the shark hidden behind the bubbles and the diver in the middle, while M2F-VIS fails to segment these instances. In the last two rows, our model successfully segments the person in different poses, while M2F-VIS fails to segment this person's arm in the first frame.