Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model

Zheng Zhang, Yeyao Ma, Enming Zhang, Xiang Bai

TL;DR

PSALM expands Large Multimodal Models to pixelwise segmentation by introducing a mask decoder and a flexible input schema that unifies images, task instructions, condition prompts, and mask tokens. A Mask Generator uses multi‑level visual features, mask tokens, and condition embeddings, guided by a bipartite matching loss $L = L_{mask} + L_{cls}$, enabling joint training across COCO Panoptic, RefCOCO variants, and COCO-Interactive, with strong zero-shot performance on open-vocabulary segmentation, generalized referring segmentation, and video object segmentation. Built on a Swin-based visual encoder and a Phi‑1.5 LLM, PSALM performs vision-language alignment before joint training, decouples mask proposal from classification for efficiency, and demonstrates robust generalization with a task-conditional prompting strategy. The work advances a GPT-like moment in computer vision by showing that a single LMM-based system can handle diverse segmentation tasks, improve cross-task performance through joint training, and generalize to unseen tasks without task-specific fine-tuning, while releasing code and pretrained models for the community.

Abstract

PSALM is a powerful extension of the Large Multi-modal Model (LMM) to address the segmentation task challenges. To overcome the limitation of the LMM being limited to textual output, PSALM incorporates a mask decoder and a well-designed input schema to handle a variety of segmentation tasks. This schema includes images, task instructions, conditional prompts, and mask tokens, which enable the model to generate and classify segmentation masks effectively. The flexible design of PSALM supports joint training across multiple datasets and tasks, leading to improved performance and task generalization. PSALM achieves superior results on several benchmarks, such as RefCOCO/RefCOCO+/RefCOCOg, COCO Panoptic Segmentation, and COCO-Interactive, and further exhibits zero-shot capabilities on unseen tasks, such as open-vocabulary segmentation, generalized referring expression segmentation and video object segmentation, making a significant step towards a GPT moment in computer vision. Through extensive experiments, PSALM demonstrates its potential to transform the domain of image segmentation, leveraging the robust visual understanding capabilities of LMMs as seen in natural language processing. Code and models are available at https://github.com/zamling/PSALM.

PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model

TL;DR

PSALM expands Large Multimodal Models to pixelwise segmentation by introducing a mask decoder and a flexible input schema that unifies images, task instructions, condition prompts, and mask tokens. A Mask Generator uses multi‑level visual features, mask tokens, and condition embeddings, guided by a bipartite matching loss , enabling joint training across COCO Panoptic, RefCOCO variants, and COCO-Interactive, with strong zero-shot performance on open-vocabulary segmentation, generalized referring segmentation, and video object segmentation. Built on a Swin-based visual encoder and a Phi‑1.5 LLM, PSALM performs vision-language alignment before joint training, decouples mask proposal from classification for efficiency, and demonstrates robust generalization with a task-conditional prompting strategy. The work advances a GPT-like moment in computer vision by showing that a single LMM-based system can handle diverse segmentation tasks, improve cross-task performance through joint training, and generalize to unseen tasks without task-specific fine-tuning, while releasing code and pretrained models for the community.

Abstract

PSALM is a powerful extension of the Large Multi-modal Model (LMM) to address the segmentation task challenges. To overcome the limitation of the LMM being limited to textual output, PSALM incorporates a mask decoder and a well-designed input schema to handle a variety of segmentation tasks. This schema includes images, task instructions, conditional prompts, and mask tokens, which enable the model to generate and classify segmentation masks effectively. The flexible design of PSALM supports joint training across multiple datasets and tasks, leading to improved performance and task generalization. PSALM achieves superior results on several benchmarks, such as RefCOCO/RefCOCO+/RefCOCOg, COCO Panoptic Segmentation, and COCO-Interactive, and further exhibits zero-shot capabilities on unseen tasks, such as open-vocabulary segmentation, generalized referring expression segmentation and video object segmentation, making a significant step towards a GPT moment in computer vision. Through extensive experiments, PSALM demonstrates its potential to transform the domain of image segmentation, leveraging the robust visual understanding capabilities of LMMs as seen in natural language processing. Code and models are available at https://github.com/zamling/PSALM.
Paper Structure (27 sections, 1 equation, 12 figures, 18 tables)

This paper contains 27 sections, 1 equation, 12 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Large Multimodal Models
  4. Pixel Reasoning with LMM
  5. Unified Segmentation Framework
  6. Methods
  7. Large Multimodal Model and Input Schema
  8. Mask Generator
  9. Design of Condition Prompts
  10. Training Objectives
  11. Experiments
  12. Datasets and Tasks
  13. Implementation Details
  14. Ablations
  15. System-Level Comparison on In-Domain Tasks
  16. ...and 12 more sections

Figures (12)

  • Figure 1: PSALM has capability to handle multiple segmentation tasks in only one single model. We visualize some tasks, including Panoptic segmentation in COCO coco; Open-Vocabulary instance segmentation in ADE20K ade20k; Interactive segmentation in COCO-Interactive; Referring segmentation in RefCOCO refcoco; Generalized referring segmentation in gRefCOCO grefcoco; Ego-exo correspondence in Ego-Exo4d egoexo; Video object segmentation in DAVIS2017 davis.
  • Figure 2: PSALM architecture overview.
  • Figure 3: Detailed processing for different condition prompts. (a) shows the processing for category condition. (b) shows the processing for sentence condition. (c) shows the processing for visual-prior condition.
  • Figure 4: Visualization of different types of visual prompts
  • Figure 5: More examples of panoptic segmentation in COCO coco.
  • ...and 7 more figures