Imagining the Unseen: Generative Location Modeling for Object Placement

TL;DR

This work tackles the problem of locating plausible placements for non existing objects in a scene by proposing a generative location model. The model conditions on the input image and object class and autoregressively predicts bounding box coordinates, effectively handling the one to many nature of plausible placements and data sparsity. It leverages negative labels through Direct Preference Optimization to refine predictions, and demonstrates superior placement accuracy on the OPA dataset as well as improved realism in downstream object insertion tasks when paired with inpainting. The results indicate that explicit location modeling enhances both the quality and coherence of inserted objects, suggesting broad utility for automated content creation, data synthesis, and planning in robotics and VR. The approach also shows flexibility to extend to richer spatial representations such as depth or 3D layouts in the future.

Abstract

Location modeling, or determining where non-existing objects could feasibly appear in a scene, has the potential to benefit numerous computer vision tasks, from automatic object insertion to scene creation in virtual reality. Yet, this capability remains largely unexplored to date. In this paper, we develop a generative location model that, given an object class and an image, learns to predict plausible bounding boxes for such an object. Our approach first tokenizes the image and target object class, then decodes bounding box coordinates through an autoregressive transformer. This formulation effectively addresses two core challenges in locatio modeling: the inherent one-to-many nature of plausible locations, and the sparsity of existing location modeling datasets, where fewer than 1% of valid placements are labeled. Furthermore, we incorporate Direct Preference Optimization to leverage negative labels, refining the spatial predictions. Empirical evaluations reveal that our generative location model achieves superior placement accuracy on the OPA dataset as compared to discriminative baselines and image composition approaches. We further test our model in the context of object insertion, where it proposes locations for an off-the-shelf inpainting model to render objects. In this respect, our proposal exhibits improved visual coherence relative to state-of-the-art instruction-tuned editing methods, demonstrating a high-performing location model's utility in a downstream application.

Figures (16)

• Figure 1: A location model identifies plausible "empty spaces" for a hypothetical object, given the name of the object. Images taken from the OPA dataset.
• Figure 2: Training scheme of our generative location model during pretraining (left) and direct preference optimization (right).
• Figure 3: Annotation format for the PIPEand OPA dataset. The PIPE dataset has one groundtruth location per image, whereas OPA provides multiple positive and negative locations.
• Figure 4: Example evaluation scenario. Predicted boxes are counted only if a positive or negative ground-truth box meets an IoU above the threshold.
• Figure 5: TPR-FPR curves. Each line is constructed by sampling $\{ 10, 20, \ldots, 100 \}$ locations. Top-left is better.