Holistic Order Prediction in Natural Scenes

TL;DR

InstaFormer, a network capable of holistic order prediction, is proposed, solely given an input RGB image, which returns the full occlusion and depth orderings for all the instances in the scene in a single forward pass.

Abstract

Even in controlled settings, understanding instance-wise geometries is a challenging task for a wide range of visual models. Although specialized systems exist, modern arts rely on expensive input formats (category labels, binary segmentation masks) and inference costs (a quadratic amount of forward passes). We mitigate these limitations by proposing InstaFormer, a network capable of holistic order prediction. That is, solely given an input RGB image, InstaFormer returns the full occlusion and depth orderings for all the instances in the scene in a single forward pass. At its core, InstaFormer relies on interactions between object queries and latent mask descriptors that semantically represent the same objects while carrying complementary information. We comprehensively benchmark and ablate our approach to highlight its effectiveness. Our code and models are open-source and available at this URL: https://github.com/SNU-VGILab/InstaOrder.

Figures (11)

• Figure 1: In-the-wild inferences using InstaFormero,d. We display the segmentation results along the occlusion and depth order predictions of images gathered from the web. Instances are represented as nodes in the graphs. Edges are arrows characterizing their order relations. InstaFormer provides accurate instance-wise geometries. Moreover, converting its outputs to text format enables VLMs to understand geometries in a zero-shot manner better.
• Figure 2: Overview of our holistic approach. We compare the holistic approach to other inference strategies (Fig. \ref{['fig:order_arch']}) and highlight the input-output format discrepancies (Tab. \ref{['tab:in_out_cost']}).
• Figure 3: Overview of InstaFormer. The architecture comprises two modules. The first module generates the mask embeddings $Q$, the per-pixel embedding $P$, and the masks $M$. In practice, we use Mask2Former cheng2022masked. Then, a transformer-based geometrical order module predicts the orders from these three inputs. InstaFormer is capable of end-to-end holistic geometrical order predictions.
• Figure 4: Qualitative results obtained using our holistic InstaFormero,d network. The first row showcases in-the-wild examples from images gathered on the web. The remaining rows are extracted from the validation set of InstaOrder. We show the segmentation predictions and represent their corresponding predicted occlusion and depth matrices as ordering graphs juxtaposed to them.
• Figure 5: Inference cost. We benchmark the runtime and memory cost of InstaOrderNeto,d and our InstaFormero,d. All measures are recorded on a single NVIDIA RTX A6000.