Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Path-Adaptive Matting for Efficient Inference Under Various Computational Cost Constraints

Qinglin Liu, Zonglin Li, Xiaoqian Lv, Xin Sun, Ru Li, Shengping Zhang

TL;DR

This work addresses efficient image matting under varying computational cost constraints by introducing Path-Adaptive Matting (PAM). PAM formulates training as a bilevel optimization that jointly learns a matting network and a path estimator within a unified, cost-aware architecture, enabling dynamic path selection conditioned on image context and FLOP limits. The framework integrates a cost-constraint embedding, a path-learning structure with path selection and learnable connect layers, and a performance-aware online path-learning strategy that labels paths using a prior distribution and network predictions. Empirical results on five matting datasets show PAM achieves competitive accuracy across mild to aggressive cost constraints while significantly reducing compute and parameter counts, with strong generalization to real-world data. The combination of bilevel optimization, cost-aware path design, and online path labeling presents a practical approach to deploy high-quality matting on diverse devices and hardware.

Abstract

In this paper, we explore a novel image matting task aimed at achieving efficient inference under various computational cost constraints, specifically FLOP limitations, using a single matting network. Existing matting methods which have not explored scalable architectures or path-learning strategies, fail to tackle this challenge. To overcome these limitations, we introduce Path-Adaptive Matting (PAM), a framework that dynamically adjusts network paths based on image contexts and computational cost constraints. We formulate the training of the computational cost-constrained matting network as a bilevel optimization problem, jointly optimizing the matting network and the path estimator. Building on this formalization, we design a path-adaptive matting architecture by incorporating path selection layers and learnable connect layers to estimate optimal paths and perform efficient inference within a unified network. Furthermore, we propose a performance-aware path-learning strategy to generate path labels online by evaluating a few paths sampled from the prior distribution of optimal paths and network estimations, enabling robust and efficient online path learning. Experiments on five image matting datasets demonstrate that the proposed PAM framework achieves competitive performance across a range of computational cost constraints.

Path-Adaptive Matting for Efficient Inference Under Various Computational Cost Constraints

TL;DR

This work addresses efficient image matting under varying computational cost constraints by introducing Path-Adaptive Matting (PAM). PAM formulates training as a bilevel optimization that jointly learns a matting network and a path estimator within a unified, cost-aware architecture, enabling dynamic path selection conditioned on image context and FLOP limits. The framework integrates a cost-constraint embedding, a path-learning structure with path selection and learnable connect layers, and a performance-aware online path-learning strategy that labels paths using a prior distribution and network predictions. Empirical results on five matting datasets show PAM achieves competitive accuracy across mild to aggressive cost constraints while significantly reducing compute and parameter counts, with strong generalization to real-world data. The combination of bilevel optimization, cost-aware path design, and online path labeling presents a practical approach to deploy high-quality matting on diverse devices and hardware.

Abstract

In this paper, we explore a novel image matting task aimed at achieving efficient inference under various computational cost constraints, specifically FLOP limitations, using a single matting network. Existing matting methods which have not explored scalable architectures or path-learning strategies, fail to tackle this challenge. To overcome these limitations, we introduce Path-Adaptive Matting (PAM), a framework that dynamically adjusts network paths based on image contexts and computational cost constraints. We formulate the training of the computational cost-constrained matting network as a bilevel optimization problem, jointly optimizing the matting network and the path estimator. Building on this formalization, we design a path-adaptive matting architecture by incorporating path selection layers and learnable connect layers to estimate optimal paths and perform efficient inference within a unified network. Furthermore, we propose a performance-aware path-learning strategy to generate path labels online by evaluating a few paths sampled from the prior distribution of optimal paths and network estimations, enabling robust and efficient online path learning. Experiments on five image matting datasets demonstrate that the proposed PAM framework achieves competitive performance across a range of computational cost constraints.

Paper Structure

This paper contains 16 sections, 12 equations, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Problem Formulation
  5. Path-Adaptive Matting Architecture
  6. Cost Constraint Embedding.
  7. Path Learning Structure.
  8. Performance-Aware Path Learning
  9. Prior Distribution Estimation.
  10. Online Path Label Generation.
  11. Loss Functions
  12. Experiments
  13. Implementation details
  14. Results on the Synthetic Datasets
  15. Results on the Real-world Dataset
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Overview of the path-adaptive matting architecture. We build a lightweight matting backbone using regular and depthwise convolution layers. To enable path-adaptive inference, a path-learning structure is introduced, which uses path selection layers to estimate network paths based on cost constraints and image context, and learnable connect layers for layer bypassing.
  • Figure 2: Qualitative results on the Adobe Composition-1K dataset. (a) Input Image. (b) Trimap. (c) Ground Truth. (d) GCAMatting. (e) MatteFormer. (f) PAM (Aggressive). (g) PAM (Moderate). (h) PAM (Mild).