Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Training-Free Out-Of-Distribution Segmentation With Foundation Models

Laith Nayal, Hadi Salloum, Ahmad Taha, Yaroslav Kholodov, Alexander Gasnikov

TL;DR

This work tackles training-free OoD segmentation by leveraging foundation-model features (InternImage) and a simple clustering-based pipeline. By clustering deep features with K-Means and aligning clusters to a per-pixel max-logit confidence map, the method identifies OoD regions without any outlier data or fine-tuning. It achieves competitive OoD segmentation results on RoadAnomaly (AP ≈ 50.02) and ADE-OoD (AP ≈ 48.77 with InternImage-L), illustrating robust generalization across distinct distribution shifts. The findings highlight the potential of minimal-assumption approaches for open-world segmentation, with future work aiming to improve upsampling and broaden applicability to existing segmentation architectures.

Abstract

Detecting unknown objects in semantic segmentation is crucial for safety-critical applications such as autonomous driving. Large vision foundation models, including DINOv2, InternImage, and CLIP, have advanced visual representation learning by providing rich features that generalize well across diverse tasks. While their strength in closed-set semantic tasks is established, their capability to detect out-of-distribution (OoD) regions in semantic segmentation remains underexplored. In this work, we investigate whether foundation models fine-tuned on segmentation datasets can inherently distinguish in-distribution (ID) from OoD regions without any outlier supervision. We propose a simple, training-free approach that utilizes features from the InternImage backbone and applies K-Means clustering alongside confidence thresholding on raw decoder logits to identify OoD clusters. Our method achieves 50.02 Average Precision on the RoadAnomaly benchmark and 48.77 on the benchmark of ADE-OoD with InternImage-L, surpassing several supervised and unsupervised baselines. These results suggest a promising direction for generic OoD segmentation methods that require minimal assumptions or additional data.

Training-Free Out-Of-Distribution Segmentation With Foundation Models

TL;DR

This work tackles training-free OoD segmentation by leveraging foundation-model features (InternImage) and a simple clustering-based pipeline. By clustering deep features with K-Means and aligning clusters to a per-pixel max-logit confidence map, the method identifies OoD regions without any outlier data or fine-tuning. It achieves competitive OoD segmentation results on RoadAnomaly (AP ≈ 50.02) and ADE-OoD (AP ≈ 48.77 with InternImage-L), illustrating robust generalization across distinct distribution shifts. The findings highlight the potential of minimal-assumption approaches for open-world segmentation, with future work aiming to improve upsampling and broaden applicability to existing segmentation architectures.

Abstract

Detecting unknown objects in semantic segmentation is crucial for safety-critical applications such as autonomous driving. Large vision foundation models, including DINOv2, InternImage, and CLIP, have advanced visual representation learning by providing rich features that generalize well across diverse tasks. While their strength in closed-set semantic tasks is established, their capability to detect out-of-distribution (OoD) regions in semantic segmentation remains underexplored. In this work, we investigate whether foundation models fine-tuned on segmentation datasets can inherently distinguish in-distribution (ID) from OoD regions without any outlier supervision. We propose a simple, training-free approach that utilizes features from the InternImage backbone and applies K-Means clustering alongside confidence thresholding on raw decoder logits to identify OoD clusters. Our method achieves 50.02 Average Precision on the RoadAnomaly benchmark and 48.77 on the benchmark of ADE-OoD with InternImage-L, surpassing several supervised and unsupervised baselines. These results suggest a promising direction for generic OoD segmentation methods that require minimal assumptions or additional data.

Paper Structure

This paper contains 23 sections, 3 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Feature Extraction
  5. Clustering
  6. Clustering:
  7. Logits Extraction
  8. Max Logits for Confidence Estimation
  9. Cluster Classification Using Logits
  10. Experiments and Analysis
  11. Dataset
  12. Experimental Setup
  13. Model Sizes and Architectures
  14. Hyperparameters
  15. Evaluation Metrics
  16. ...and 8 more sections

Figures (5)

  • Figure 1: Input image (top-left), Ground Truth (top-right), Clusters (bottom-left), Ours (bottom-right).
  • Figure 2: Proposed training-free OoD segmentation pipeline: Input images are processed by the InternImage backbone for feature extraction and upsampling. Simultaneously, UPerNet decodes semantic logits, generating max-logit confidence heatmaps. Features are clustered via K-Means, and cluster classification identifies OoD groups (gray cluster). Final OoD masks are compared against ground truth.
  • Figure 3: (a) Hierarchical architecture with stem/downsampling layers; (b) DCNv3 block: LN $\rightarrow$ deformable conv $\rightarrow$ FFN; (c) Multi-group sampling adapts to object structure.
  • Figure 4: Qualitative segmentation results (Set 1).
  • Figure 5: Qualitative segmentation results (Set 2).