Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hyper-Transformer for Amodal Completion

Jianxiong Gao, Xuelin Qian, Longfei Liang, Junwei Han, Yanwei Fu

TL;DR

This work addresses amodal completion by introducing H-TAN, a dual-branch framework that learns object shape priors from background information using a transformer-based hypernetwork. The Hyper Transformer generates instance-specific weights for a Dynamic Head that, guided by mask features, predicts precise amodal masks without relying on extra information or multi-stage training. Across three benchmarks (KINS, COCOA-cls, D2SA), H-TAN achieves state-of-the-art results, with ablations confirming the critical roles of the hyper transformer, multi-scale fusion, and skip connections. The approach offers a scalable, efficient pathway for robust amodal segmentation with potential impact on autonomous driving, robotics, and AR applications.

Abstract

Amodal object completion is a complex task that involves predicting the invisible parts of an object based on visible segments and background information. Learning shape priors is crucial for effective amodal completion, but traditional methods often rely on two-stage processes or additional information, leading to inefficiencies and potential error accumulation. To address these shortcomings, we introduce a novel framework named the Hyper-Transformer Amodal Network (H-TAN). This framework utilizes a hyper transformer equipped with a dynamic convolution head to directly learn shape priors and accurately predict amodal masks. Specifically, H-TAN uses a dual-branch structure to extract multi-scale features from both images and masks. The multi-scale features from the image branch guide the hyper transformer in learning shape priors and in generating the weights for dynamic convolution tailored to each instance. The dynamic convolution head then uses the features from the mask branch to predict precise amodal masks. We extensively evaluate our model on three benchmark datasets: KINS, COCOA-cls, and D2SA, where H-TAN demonstrated superior performance compared to existing methods. Additional experiments validate the effectiveness and stability of the novel hyper transformer in our framework.

Hyper-Transformer for Amodal Completion

TL;DR

This work addresses amodal completion by introducing H-TAN, a dual-branch framework that learns object shape priors from background information using a transformer-based hypernetwork. The Hyper Transformer generates instance-specific weights for a Dynamic Head that, guided by mask features, predicts precise amodal masks without relying on extra information or multi-stage training. Across three benchmarks (KINS, COCOA-cls, D2SA), H-TAN achieves state-of-the-art results, with ablations confirming the critical roles of the hyper transformer, multi-scale fusion, and skip connections. The approach offers a scalable, efficient pathway for robust amodal segmentation with potential impact on autonomous driving, robotics, and AR applications.

Abstract

Amodal object completion is a complex task that involves predicting the invisible parts of an object based on visible segments and background information. Learning shape priors is crucial for effective amodal completion, but traditional methods often rely on two-stage processes or additional information, leading to inefficiencies and potential error accumulation. To address these shortcomings, we introduce a novel framework named the Hyper-Transformer Amodal Network (H-TAN). This framework utilizes a hyper transformer equipped with a dynamic convolution head to directly learn shape priors and accurately predict amodal masks. Specifically, H-TAN uses a dual-branch structure to extract multi-scale features from both images and masks. The multi-scale features from the image branch guide the hyper transformer in learning shape priors and in generating the weights for dynamic convolution tailored to each instance. The dynamic convolution head then uses the features from the mask branch to predict precise amodal masks. We extensively evaluate our model on three benchmark datasets: KINS, COCOA-cls, and D2SA, where H-TAN demonstrated superior performance compared to existing methods. Additional experiments validate the effectiveness and stability of the novel hyper transformer in our framework.
Paper Structure (17 sections, 11 equations, 6 figures, 5 tables)

This paper contains 17 sections, 11 equations, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Amodal Instance Segmentation.
  4. Hyper Network
  5. Methodology
  6. Dual Branches
  7. Hyper Transformer
  8. Dynamic Head
  9. Experiments
  10. Experiments Details
  11. Experiments Results
  12. Discussion on the Hyper Transformer
  13. Ablation Study
  14. Conclusion
  15. Appendix / supplemental material
  16. ...and 2 more sections

Figures (6)

  • Figure 1: Visualization of Amodal Mask Quality Produced by Our Model. We showcase the amodal masks of objects annotated in the KINS and COCOA-cls datasets.
  • Figure 2: Overview of the H-TAN framework. H-TAN extracts comprehensive features from both the image and mask through two branches. The image branch uses the mask to focus on the object that needs completion, extracting multi-scale features. The mask branch receives these multi-scale features from the image. Next, we design a Hyper Transformer and a Dynamic Head. The Hyper Transformer generates the parameters for the Dynamic Head guided by the image features, ultimately predicting the precise $M_a$.
  • Figure 3: Details of the use of learnable tokens in the hyper transformer as weights for the dynamic head, enabling precise prediction of the amodal mask based on mask features.
  • Figure 4: Qualitative results compared with VRSP, C2F-Seg, and our H-TAN on KINS, COCOA-cls, and D2SA datasets. VM and GT denote the ground-truth visible mask and amodal mask, respectively. Best viewed in color and zoomed in for details.
  • Figure A.1: More Qualitative results compared with VRSP, C2F-Seg, and our model.
  • ...and 1 more figures