Learning Concept-Based Causal Transition and Symbolic Reasoning for Visual Planning

TL;DR

An interpretable and generalizable visual planning framework consisting of a novel Substitution-based Concept Learner that abstracts visual inputs into disentangled concept representations, symbol abstraction and reasoning that performs task planning via the learned symbols and a Visual Causal Transition model that grounds visual causal transitions to semantically similar real-world actions is proposed.

Abstract

Visual planning simulates how humans make decisions to achieve desired goals in the form of searching for visual causal transitions between an initial visual state and a final visual goal state. It has become increasingly important in egocentric vision with its advantages in guiding agents to perform daily tasks in complex environments. In this paper, we propose an interpretable and generalizable visual planning framework consisting of i) a novel Substitution-based Concept Learner (SCL) that abstracts visual inputs into disentangled concept representations, ii) symbol abstraction and reasoning that performs task planning via the self-learned symbols, and iii) a Visual Causal Transition model (ViCT) that grounds visual causal transitions to semantically similar real-world actions. Given an initial state, we perform goal-conditioned visual planning with a symbolic reasoning method fueled by the learned representations and causal transitions to reach the goal state. To verify the effectiveness of the proposed model, we collect a large-scale visual planning dataset based on AI2-THOR, dubbed as CCTP. Extensive experiments on this challenging dataset demonstrate the superior performance of our method in visual task planning. Empirically, we show that our framework can generalize to unseen task trajectories, unseen object categories, and real-world data. Further details of this work are provided at https://fqyqc.github.io/ConTranPlan/.

Figures (6)

• Figure 1: Our visual planning framework. Given an initial state and a goal state, we aim to predict the intermediate states (in the second row) that will guide a robot to manipulate the target objects (in the third row). The disentangled concept-based representation $C$ and abstracted symbol representation $\Omega$, as well as their corresponding causal transition $\mathcal{T}$ and symbol reasoning $S$, are effectively combined into a bi-level planning framework for better generalization (in the first row).
• Figure 2: Architecture of SCL. Foreground images $X_{0,f}$ and $X_{1,f}$ differ only in the COLOR concept. After extracting their concept tokens and assuming the token $c^5_i$ to represent the color concept, the COLOR concept $c_0^5$ of $X_{0,f}$ is substituted by $c_1^5$ from $X_{1,f}$, which are then fed into the detokenizer and decoder to reconstruct images.
• Figure 3: Architecture of ViCT. The concept tokenizer extracts object concept tokens for causal transition. The causal transition model transforms concept tokens from $C_0$ to $C_1^{'}$ with the action embedding $\mathcal{V}(a)$. The background encoder converts the background image into latent vectors, which are then combined with predicted concept tokens $C_1^{'}$ to generate the effect image $\Tilde X_1$.
• Figure 4: Qualitative results of our visual planning model. The top two samples are obtained from the level-3 dataset, and the middle two are from the level-4 dataset. Our model demonstrates its ability to manage tasks of varying lengths, effectively plan action sequences, and generate intermediate and goal state images. Notably, the first sample from the level-4 dataset generates a different path than the ground truth but still achieves success and maintains high efficiency. The bottom two samples are from our real-world data experiments, corresponding to the level-3 dataset. To simplify implementation, we focus on visually planning the target objects in the real-world images, ignoring the background.
• Figure 5: Fine-grained attribute-level concept manipulation. The concept learner generates new images by substituting each concept token $c_0^{i}$ from $X_{0,f}$ with $c_1^{i}$ from $X_{1,f}$.