HAPFI: History-Aware Planning based on Fused Information

TL;DR

HAPFI addresses Embodied Instruction Following by leveraging historical information across multiple modalities through deep, early fusion and cross-modal attention. It combines historical visual features (RGB and bounding boxes) with linguistic context (high-level instructions and sub-goal history) via Mutually Attentive Fusion, followed by sub-goal classification to predict next actions, objects, and receptacles. Empirical results on ALFRED show that full multi-modal history integration yields superior planning performance and robustness to unseen environments, with qualitative analysis illustrating effective re-planning after intermediate failures. This approach advances EIF by enabling more informed, context-aware planning that adapts to real-world uncertainties and changes.

Abstract

Embodied Instruction Following (EIF) is a task of planning a long sequence of sub-goals given high-level natural language instructions, such as "Rinse a slice of lettuce and place on the white table next to the fork". To successfully execute these long-term horizon tasks, we argue that an agent must consider its past, i.e., historical data, when making decisions in each step. Nevertheless, recent approaches in EIF often neglects the knowledge from historical data and also do not effectively utilize information across the modalities. To this end, we propose History-Aware Planning based on Fused Information (HAPFI), effectively leveraging the historical data from diverse modalities that agents collect while interacting with the environment. Specifically, HAPFI integrates multiple modalities, including historical RGB observations, bounding boxes, sub-goals, and high-level instructions, by effectively fusing modalities via our Mutually Attentive Fusion method. Through experiments with diverse comparisons, we show that an agent utilizing historical multi-modal information surpasses all the compared methods that neglect the historical data in terms of action planning capability, enabling the generation of well-informed action plans for the next step. Moreover, we provided qualitative evidence highlighting the significance of leveraging historical multi-modal data, particularly in scenarios where the agent encounters intermediate failures, showcasing its robust re-planning capabilities.

Figures (3)

• Figure 1: Planning Procedure of the EIF Task Employing HAPFI. The agent utilizes visual and sub-goal history as demonstrated in the gray box. It determines its next action by referencing this historical information, the current RGB observation, and the provided high-level instruction. This decision-making process is repeated in a continuous plan-action loop until the agent decides the completion of the entire task.
• Figure 2: Overview of History-aware planning based on Fusion Integration Image history $O_{<n}$ and current image $O_{n}$ are encoded through a pretrained Vision Transformer (ViT) while bounding bbox (bbox) $B_{<n}$ history and current bbox $B_{n}$ are encoded using ViT trained from scratch. Sub-goal history $S_{<n}$ and high-level instruction $I$ are encoded by BERT architecture trained from scratch. The information encoded in each modality is integrated through the fusion module and then predicts actions $S^a_{n}$, objects $S^o_{n}$, and receptacles $S^r_{n}$ separately through task-specific heads. c denotes concatenation, and + denotes element-wise sum.
• Figure 3: Illustration of re-planning procedure This figure illustrates the re-planning procedure. In both rows, the agent performs four actions following the high-level instruction: picking up the knife, navigating to the bowl, putting the knife in the bowl, and picking up the bowl with the knife. In the first row, an erroneous navigation occurs, leading the agent to the microwave. In the second row, a manipulation error results in the agent dropping the bowl and knife.