Adapting Interleaved Encoders with PPO for Language-Guided Reinforcement Learning in BabyAI

TL;DR

This paper tackles the challenge of reinforcement learning tasks that require joint visual and language understanding, where perception and policy are typically decoupled. It introduces the Perception–Decision Interleaving Transformer (PDiT) architecture, integrated with a PPO policy and a CLIP‑style contrastive loss, for language‑guided navigation in BabyAI GoToLocal. The total objective is $\mathcal{L}_{total}=\mathcal{L}_{PPO}+\lambda_1\mathcal{L}_{CLIP}+\lambda_2\mathcal{L}_{sup}$ with joint gradient coupling that enables direct learning signals from policy to perception. Empirically, PDiT–PPO with multimodal alignment achieves more stable convergence, reduced reward variance (e.g., $42\%$), and stronger alignment between textual missions and visual cues compared to a standard PPO baseline, indicating that tightly integrated, multimodal interleaving can enhance robustness and sample efficiency in language‑guided autonomous agents.

Abstract

Deep reinforcement learning agents often struggle when tasks require understanding both vision and language. Conventional architectures typically isolate perception (for example, CNN-based visual encoders) from decision-making (policy networks). This separation can be inefficient, since the policy's failures do not directly help the perception module learn what is important. To address this, we implement the Perception-Decision Interleaving Transformer (PDiT) architecture introduced by Mao et al. (2023), a model that alternates between perception and decision layers within a single transformer. This interleaving allows feedback from decision-making to refine perceptual features dynamically. In addition, we integrate a contrastive loss inspired by CLIP to align textual mission embeddings with visual scene features. We evaluate the PDiT encoders on the BabyAI GoToLocal environment and find that the approach achieves more stable rewards and stronger alignment compared to a standard PPO baseline. The results suggest that interleaved transformer encoders are a promising direction for developing more integrated autonomous agents.

Figures (2)

• Figure 1: Conceptual diagram of Perception–Decision Interleaving Transformer showing visual-text input fusion, interleaved layers, and joint optimization paths.
• Figure 2: BabyAI GoToLocal environment. The agent perceives a 7×7 field of view and receives mission instructions as text.