Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Multiple Representations with Inconsistency-Guided Detail Regularization for Mask-Guided Matting

Weihao Jiang, Zhaozhi Xie, Yuxiang Lu, Longjie Qi, Jingyong Cai, Hiroyuki Uchiyama, Bin Chen, Yue Ding, Hongtao Lu

TL;DR

A novel auxiliary learning framework for mask-guided matting models is proposed, incorporating three auxiliary tasks: semantic segmentation, edge detection, and background line detection besides matting, to learn different and effective representations from different types of data and annotations.

Abstract

Mask-guided matting networks have achieved significant improvements and have shown great potential in practical applications in recent years. However, simply learning matting representation from synthetic and lack-of-real-world-diversity matting data, these approaches tend to overfit low-level details in wrong regions, lack generalization to objects with complex structures and real-world scenes such as shadows, as well as suffer from interference of background lines or textures. To address these challenges, in this paper, we propose a novel auxiliary learning framework for mask-guided matting models, incorporating three auxiliary tasks: semantic segmentation, edge detection, and background line detection besides matting, to learn different and effective representations from different types of data and annotations. Our framework and model introduce the following key aspects: (1) to learn real-world adaptive semantic representation for objects with diverse and complex structures under real-world scenes, we introduce extra semantic segmentation and edge detection tasks on more diverse real-world data with segmentation annotations; (2) to avoid overfitting on low-level details, we propose a module to utilize the inconsistency between learned segmentation and matting representations to regularize detail refinement; (3) we propose a novel background line detection task into our auxiliary learning framework, to suppress interference of background lines or textures. In addition, we propose a high-quality matting benchmark, Plant-Mat, to evaluate matting methods on complex structures. Extensively quantitative and qualitative results show that our approach outperforms state-of-the-art mask-guided methods.

Learning Multiple Representations with Inconsistency-Guided Detail Regularization for Mask-Guided Matting

TL;DR

A novel auxiliary learning framework for mask-guided matting models is proposed, incorporating three auxiliary tasks: semantic segmentation, edge detection, and background line detection besides matting, to learn different and effective representations from different types of data and annotations.

Abstract

Mask-guided matting networks have achieved significant improvements and have shown great potential in practical applications in recent years. However, simply learning matting representation from synthetic and lack-of-real-world-diversity matting data, these approaches tend to overfit low-level details in wrong regions, lack generalization to objects with complex structures and real-world scenes such as shadows, as well as suffer from interference of background lines or textures. To address these challenges, in this paper, we propose a novel auxiliary learning framework for mask-guided matting models, incorporating three auxiliary tasks: semantic segmentation, edge detection, and background line detection besides matting, to learn different and effective representations from different types of data and annotations. Our framework and model introduce the following key aspects: (1) to learn real-world adaptive semantic representation for objects with diverse and complex structures under real-world scenes, we introduce extra semantic segmentation and edge detection tasks on more diverse real-world data with segmentation annotations; (2) to avoid overfitting on low-level details, we propose a module to utilize the inconsistency between learned segmentation and matting representations to regularize detail refinement; (3) we propose a novel background line detection task into our auxiliary learning framework, to suppress interference of background lines or textures. In addition, we propose a high-quality matting benchmark, Plant-Mat, to evaluate matting methods on complex structures. Extensively quantitative and qualitative results show that our approach outperforms state-of-the-art mask-guided methods.
Paper Structure (13 sections, 6 equations, 8 figures, 6 tables)

This paper contains 13 sections, 6 equations, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Our Auxiliary Learning Framework
  4. Learning real-world adaptive semantic representation
  5. Inconsistency-Guided detail regularization
  6. Learning background line detection
  7. Our Plant-Mat Benchmark
  8. Experiments
  9. Implementation details
  10. Benchmarks
  11. Qualitative and quantitative comparisons
  12. Ablation analysis
  13. Conclusion

Figures (8)

  • Figure 1: Qualitative comparisons between MGMatting mgm_ref and Ours. From left to right, the input image and a binary guidance mask, MGMatting, Ours.
  • Figure 2: Visualization of the inconsistency between matting and segmentation masks, which points out important low-level details.
  • Figure 3: Overview of our proposed auxiliary learning framework and our proposed network. The proposed network leans multiple representations from different types of data and annotations in our auxiliary learning framework. Our IGDR module uses the inconsistency between matting representation and real-world adaptive semantic representation to regularize refinement on low-level details.
  • Figure 4: Visualization of a training sample for background line detection.
  • Figure 5: The qualitative comparisons between MGMatting mgm_ref and ours on various real-world images from test sets aimmgm_refAM-2k.
  • ...and 3 more figures