Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Bottom-Up Approach to Class-Agnostic Image Segmentation

Sebastian Dille, Ari Blondal, Sylvain Paris, Yağız Aksoy

TL;DR

This work presents a novel bottom-up formulation for addressing the class-agnostic segmentation problem, employing losses inspired by metric learning literature as well as losses defined in a novel segmentation-space representation, and demonstrates exceptional generalization capability.

Abstract

Class-agnostic image segmentation is a crucial component in automating image editing workflows, especially in contexts where object selection traditionally involves interactive tools. Existing methods in the literature often adhere to top-down formulations, following the paradigm of class-based approaches, where object detection precedes per-object segmentation. In this work, we present a novel bottom-up formulation for addressing the class-agnostic segmentation problem. We supervise our network directly on the projective sphere of its feature space, employing losses inspired by metric learning literature as well as losses defined in a novel segmentation-space representation. The segmentation results are obtained through a straightforward mean-shift clustering of the estimated features. Our bottom-up formulation exhibits exceptional generalization capability, even when trained on datasets designed for class-based segmentation. We further showcase the effectiveness of our generic approach by addressing the challenging task of cell and nucleus segmentation. We believe that our bottom-up formulation will offer valuable insights into diverse segmentation challenges in the literature.

A Bottom-Up Approach to Class-Agnostic Image Segmentation

TL;DR

This work presents a novel bottom-up formulation for addressing the class-agnostic segmentation problem, employing losses inspired by metric learning literature as well as losses defined in a novel segmentation-space representation, and demonstrates exceptional generalization capability.

Abstract

Class-agnostic image segmentation is a crucial component in automating image editing workflows, especially in contexts where object selection traditionally involves interactive tools. Existing methods in the literature often adhere to top-down formulations, following the paradigm of class-based approaches, where object detection precedes per-object segmentation. In this work, we present a novel bottom-up formulation for addressing the class-agnostic segmentation problem. We supervise our network directly on the projective sphere of its feature space, employing losses inspired by metric learning literature as well as losses defined in a novel segmentation-space representation. The segmentation results are obtained through a straightforward mean-shift clustering of the estimated features. Our bottom-up formulation exhibits exceptional generalization capability, even when trained on datasets designed for class-based segmentation. We further showcase the effectiveness of our generic approach by addressing the challenging task of cell and nucleus segmentation. We believe that our bottom-up formulation will offer valuable insights into diverse segmentation challenges in the literature.
Paper Structure (20 sections, 10 equations, 3 figures, 5 tables)

This paper contains 20 sections, 10 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Generating Distinguishable Features
  4. Feature Representation
  5. Determining Target Lines for each Entity
  6. Handling Imperfect Ground-truth
  7. Attraction and Repulsion
  8. Segment-space Representation
  9. Network Architecture and Training
  10. Inference-time Clustering
  11. Multi-resolution Refinement
  12. Experiments
  13. Class-agnostic Baselines
  14. Class-based Baselines
  15. Metrics
  16. ...and 5 more sections

Figures (3)

  • Figure 1: We introduce a bottom-up approach to class-agnostic image segmentation. We show that our formulation leads to generalization to images in-the-wild that are not well-represented in common training datasets. We generate detailed segmentation maps for complex scenes where other class-based or class-agnostic approaches fall short.
  • Figure 2: We show our resulting feature map in (a), reduced with PCA and colorized for visualization. Our projection into the segment space results in a set of binary maps in (b) that are compared against the ground-truth in (c) via our segmentation loss.
  • Figure 3: Two predictions from early training phases, (a) with only contrastive supervision and (b) with our segmentation loss added.