Leveraging Image Augmentation for Object Manipulation: Towards Interpretable Controllability in Object-Centric Learning

TL;DR

SlotAug tackles interpretability and interactive controllability in object-centric learning by coupling Slot Attention with an image augmentation training regime. It introduces Auxiliary Identity Manipulation (AIM) and Slot Consistency Loss (SCLoss) to promote sustainable, reversible slot manipulations, formalized through objectives like $L_{cycle}$ and $L_{total}$. A lightweight PropEnc encodes transformation instructions, enabling slot-level edits via a SlotManip module, and inference uses the Hungarian algorithm to map user intent to the closest target slot. Extensive experiments on multi-object datasets demonstrate interpretable object editing, conditional image composition, and durable slot representations in both pixel and latent spaces, with improved property-prediction performance over baselines.

Abstract

The binding problem in artificial neural networks is actively explored with the goal of achieving human-level recognition skills through the comprehension of the world in terms of symbol-like entities. Especially in the field of computer vision, object-centric learning (OCL) is extensively researched to better understand complex scenes by acquiring object representations or slots. While recent studies in OCL have made strides with complex images or videos, the interpretability and interactivity over object representation remain largely uncharted, still holding promise in the field of OCL. In this paper, we introduce a novel method, Slot Attention with Image Augmentation (SlotAug), to explore the possibility of learning interpretable controllability over slots in a self-supervised manner by utilizing an image augmentation strategy. We also devise the concept of sustainability in controllable slots by introducing iterative and reversible controls over slots with two proposed submethods: Auxiliary Identity Manipulation and Slot Consistency Loss. Extensive empirical studies and theoretical validation confirm the effectiveness of our approach, offering a novel capability for interpretable and sustainable control of object representations.

Figures (17)

• Figure 1: Overview of our method compared to the previous methods.(a) Previous methods require an additional process to manipulate slots such as feature selection during inference. (b) Our model, however, has the shared process of manipulating slots between the training and inference stages. (c) To ensure homogeneity between the training and inference stages, we incorporate scenarios involving image manipulation into the training phase. This includes the application of simple image augmentation techniques such as scaling, translating, and color shifting. (d) Upon completion of the training, our model achieves interpretable controllability, enabling users to manipulate individual objects according to their intentions.
• Figure 2: Architecture of our model. From a given image $img_{ref}$, we first generate an augmented image $img_{aug}$ (leftmost part of the figure), and the corresponding instruction $insts_{ref2aug}$ and its inverse $insts_{aug2ref}$. Our model produces slots from $img_{ref}$ and decodes the slots to reconstruct the original image ($recon_{ref}$). The slots are also manipulated with SlotManipulation module which takes $insts_{ref2aug}$ as the other input. We incorporate Auxiliary Identity Manipulation (AIM) into this manipulation process. The details are provided in the right part of the figure. The manipulated slots are then simultaneously 1) decoded into a reconstruction of the augmented image $recon_{aug}$, and 2) re-manipulated by SlotManipulation with $insts_{aug2ref}$. Our total loss consists of the reconstruction losses of reference and augmented images, and the slot-level cycle consistency.
• Figure 3: (a) Object manipulation with human-interpretable instruction. The first and second columns are the ground-truth and reconstruction images, respectively. The following columns are the results of the controls along the instructions. Here, instructions are described with the text for easy understanding. The actual instantiation of the instructions can be found in the Appendix. From the first row onwards, the results are for Tetrominoes, CLEVR, CLEVR, and PTR, respectively. (b) Conditional image composition. From given source images, we can collect specific objects, which are indicated by white numbers, and manipulate them to generate a novel image.
• Figure 4: Iterative slot manipulation. The leftmost image is the initial image from which the iterative manipulation begins. The text on each column states the instruction used for manipulation. Each row shows the results of the manipulation by v1, v2, and v3 models, respectively. Center areas are cropped for better visibility.
• Figure 5: Durability test. The leftmost image is the initial image from which the test begins. The top three rows show the results of the single-step tests where each model is instructed to alternately move the target object up and down four times each. In the multi-step test, as shown in the last row, the model performs two round-trip manipulations, each involving moving the target object down, changing its color, reverting the color, and returning the object to its original position.