Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Frequency-based Matcher for Long-tailed Semantic Segmentation

Shan Li, Lu Yang, Pu Cao, Liulei Li, Huadong Ma

TL;DR

The paper targets long-tailed semantic segmentation (LTSS) by establishing three LTSS datasets, a dual-metric evaluation system, and a transformer-based Frequency-based Matcher (FM) to address tail-class oversuppression. The FM employs one-to-many query matching with class-dependent query counts, improving supervision for low-frequency classes and achieving substantial gains over strong baselines across ADE20K-Full, COCO-Stuff-LT, and MHP-v2-LT. The work demonstrates that LTSS requires dedicated benchmarks and that FM is a practical, compatible enhancement for transformer-based segmentation models. By releasing datasets, code, and models, the authors provide a foundational platform to accelerate LTSS research and evaluation.

Abstract

The successful application of semantic segmentation technology in the real world has been among the most exciting achievements in the computer vision community over the past decade. Although the long-tailed phenomenon has been investigated in many fields, e.g., classification and object detection, it has not received enough attention in semantic segmentation and has become a non-negligible obstacle to applying semantic segmentation technology in autonomous driving and virtual reality. Therefore, in this work, we focus on a relatively under-explored task setting, long-tailed semantic segmentation (LTSS). We first establish three representative datasets from different aspects, i.e., scene, object, and human. We further propose a dual-metric evaluation system and construct the LTSS benchmark to demonstrate the performance of semantic segmentation methods and long-tailed solutions. We also propose a transformer-based algorithm to improve LTSS, frequency-based matcher, which solves the oversuppression problem by one-to-many matching and automatically determines the number of matching queries for each class. Given the comprehensiveness of this work and the importance of the issues revealed, this work aims to promote the empirical study of semantic segmentation tasks. Our datasets, codes, and models will be publicly available.

Frequency-based Matcher for Long-tailed Semantic Segmentation

TL;DR

The paper targets long-tailed semantic segmentation (LTSS) by establishing three LTSS datasets, a dual-metric evaluation system, and a transformer-based Frequency-based Matcher (FM) to address tail-class oversuppression. The FM employs one-to-many query matching with class-dependent query counts, improving supervision for low-frequency classes and achieving substantial gains over strong baselines across ADE20K-Full, COCO-Stuff-LT, and MHP-v2-LT. The work demonstrates that LTSS requires dedicated benchmarks and that FM is a practical, compatible enhancement for transformer-based segmentation models. By releasing datasets, code, and models, the authors provide a foundational platform to accelerate LTSS research and evaluation.

Abstract

The successful application of semantic segmentation technology in the real world has been among the most exciting achievements in the computer vision community over the past decade. Although the long-tailed phenomenon has been investigated in many fields, e.g., classification and object detection, it has not received enough attention in semantic segmentation and has become a non-negligible obstacle to applying semantic segmentation technology in autonomous driving and virtual reality. Therefore, in this work, we focus on a relatively under-explored task setting, long-tailed semantic segmentation (LTSS). We first establish three representative datasets from different aspects, i.e., scene, object, and human. We further propose a dual-metric evaluation system and construct the LTSS benchmark to demonstrate the performance of semantic segmentation methods and long-tailed solutions. We also propose a transformer-based algorithm to improve LTSS, frequency-based matcher, which solves the oversuppression problem by one-to-many matching and automatically determines the number of matching queries for each class. Given the comprehensiveness of this work and the importance of the issues revealed, this work aims to promote the empirical study of semantic segmentation tasks. Our datasets, codes, and models will be publicly available.
Paper Structure (17 sections, 3 equations, 4 figures, 6 tables)

This paper contains 17 sections, 3 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Semantic Segmentation
  4. Transformer-based Segmentation
  5. Long-tailed Learning
  6. Long-tailed Semantic Segmentation
  7. Problem Formulation
  8. Dataset Design
  9. Evaluation System
  10. Advanced Benchmark
  11. Experiments
  12. Implementation Details
  13. Main Results
  14. Ablation Study
  15. Analysis and Discussion
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Illustration of long-tailed semantic segmentation (LTSS) samples from the three datasets we constructed. From left to right, the frequencies are frequent, common, and rare.
  • Figure 2: The label distribution of different LTSS datasets. Figures (a)-(c) in the 1$st$ row show the long-tailed label distributions at the image level, and Figures (d)-(f) in the 2$nd$ row show the long-tailed label distributions at the pixel level.
  • Figure 3: Illustration of the proposed frequency-based matcher. (a) shows the one-to-one matcher in cheng2022masked. (b) The proposed frequency-based matcher in this paper.
  • Figure 4: Visualization of rare category predictions. We further show the image-level and pixel-level frequencies of each category. The frequent, common, and rare categories are labeled F, C, and R, respectively.