Beyond Static Perception: Integrating Temporal Context into VLMs for Cloth Folding

TL;DR

Cloth folding presents highly deformable states with frequent occlusions, making perception challenging. The paper analyzes BiFold, a LoRA-finetuned SigLIP vision-language backbone, to predict language-conditioned pick-and-place actions using a history of keyframes, specifically with a history window of $H=3$. Results show that temporal context improves state estimation, cross-modal alignment, and attention consistency, with keyframe-based history outperforming consecutive-frame variants. These findings highlight the value of implicit state representations learned from temporal context for robust deformable-object manipulation and point to future work on automatic keyframe selection and additional temporal supervision.

Abstract

Manipulating clothes is challenging due to their complex dynamics, high deformability, and frequent self-occlusions. Garments exhibit a nearly infinite number of configurations, making explicit state representations difficult to define. In this paper, we analyze BiFold, a model that predicts language-conditioned pick-and-place actions from visual observations, while implicitly encoding garment state through end-to-end learning. To address scenarios such as crumpled garments or recovery from failed manipulations, BiFold leverages temporal context to improve state estimation. We examine the internal representations of the model and present evidence that its fine-tuning and temporal context enable effective alignment between text and image regions, as well as temporal consistency.

Figures (4)

• Figure 1: Motivation: In multi-step tasks like cloth folding, the state can become visually ambiguous due to self-occlusions or compounding errors from past actions. Relying solely on the current observation can make the next action ill-defined (left). By incorporating temporal context (right), the model gains critical information enabling more accurate perception and decision-making.
• Figure 2: BiFold architecture: BiFold fine-tunes a pre-trained SigLIP siglip with LoRA hu2022lora to obtain image and text features. The same image encoder is used for the current and past observations. Each token sequence is prepended with a different learned token, and we add information about the modality and the position inside the sequence by adding a learned embedding and sinusoidal positional encodings. All tokens are concatenated and processed using a transformer encoder transformers and the output tokens of the current observation are decoded into probability distributions on the pixel space using convolutional networks.
• Figure 3: Image encoder features: While pre-trained SigLIP features offer aligned language and visual embeddings, they struggle to capture semantic correspondences across time steps. In contrast, BiFold's fine-tuned version focuses more effectively on the manipulated cloth, enhancing the model's ability to distinguish task-relevant regions critical for folding.
• Figure 4: Text$\to$Image attention: Attention scores from text to image patches to the current observation and keyframes.