LACY: A Vision-Language Model-based Language-Action Cycle for Self-Improving Robotic Manipulation

TL;DR

LACY addresses the limitations of unidirectional language-to-action policies in robotic manipulation by introducing a bidirectional language-action cycle that jointly learns L2A, A2L, and L2C within a single vision-language model. The framework uses a two-stage fine-tuning pipeline with object grounding pre-training and chain-of-thought–guided multi-task learning, combined with a confidence-based active data augmentation strategy to autonomously generate and filter training data. A self-improvement loop, powered by the L2C verifier and majority-voted sampling, yields progressively richer datasets and improved task success in both simulation and real-world pick-and-place tasks. The results demonstrate significant performance gains, enhanced grounding robustness, and reduced reliance on external labels, illustrating the potential of bidirectional grounding for scalable robot learning.

Abstract

Learning generalizable policies for robotic manipulation increasingly relies on large-scale models that map language instructions to actions (L2A). However, this one-way paradigm often produces policies that execute tasks without deeper contextual understanding, limiting their ability to generalize or explain their behavior. We argue that the complementary skill of mapping actions back to language (A2L) is essential for developing more holistic grounding. An agent capable of both acting and explaining its actions can form richer internal representations and unlock new paradigms for self-supervised learning. We introduce LACY (Language-Action Cycle), a unified framework that learns such bidirectional mappings within a single vision-language model. LACY is jointly trained on three synergistic tasks: generating parameterized actions from language (L2A), explaining observed actions in language (A2L), and verifying semantic consistency between two language descriptions (L2C). This enables a self-improving cycle that autonomously generates and filters new training data through an active augmentation strategy targeting low-confidence cases, thereby improving the model without additional human labels. Experiments on pick-and-place tasks in both simulation and the real world show that LACY improves task success rates by 56.46% on average and yields more robust language-action grounding for robotic manipulation. Project page: https://vla2026.github.io/LACY/

Figures (8)

• Figure 1: Human demonstration of toy object manipulation. Humans can readily infer task procedures from a manipulation demonstration and express them in language (e.g., "pick up the yellow block" $\rightarrow$ "place it to the right of the green block" $\rightarrow$ "grasp the blue cylinder" $\rightarrow$ "put it on the bottom right of the table"). This linguistic description enables humans to accurately replicate the demonstrated action sequence.
• Figure 2: Notations. Each demonstration $\zeta_i$ includes an image observation $\mathbf{o}_t$, a task description $\mathbf{l}_t$, and a pick-and-place action $\mathbf{a}$. The workspace is divided into a $3\times3$ grid. Coordinates $(x, y)$ are normalized to $[0,1]$, where $x,y\in[0,1]$, with $(x,y)=(0,0)$ at the left/top image border and $(x,y)=(1,1)$ at the right/bottom border.
• Figure 3: Spatial description types. Task description for placing an object uses different forms of language descriptions—absolute or relative—based on the Euclidean distance to the placing location and the proximity to the outer contour of the nearest object.
• Figure 4: Overview of the LACY framework. LACY (Language-Action CYcle) builds upon a single VLM liu2024llavanext fine-tuned to serve three roles: (1) an action generator ($\text{L2A}$), (2) an action explainer ($\text{A2L}$), and (3) a consistency verifier ($\text{L2C}$). The framework operates as a closed-loop system, where these bidirectional capabilities enable LACY to generate new high-quality training data and iteratively refine itself. (4) Each task is framed as a chain-of-thought (CoT) process, where the model first performs object grounding to predict object names and locations ($\hat{\mathcal{O}}$) and then uses this contextual information to complete the target task. (5) As shown in Algorithm \ref{['alg:checkman']}, new samples are generated and merged with the initial dataset.
• Figure 5: Binary confidence extraction from VLM outputs. The logits $z_0$ and $z_1$ corresponding to the tokens "0" and "1" are used to compute a confidence score $c$.