Action-Sketcher: From Reasoning to Action via Visual Sketches for Long-Horizon Robotic Manipulation

TL;DR

Action-Sketcher addresses the brittleness of long-horizon robotic manipulation by introducing Visual Sketch, an explicit, editable spatial intermediate that grounds language in scene geometry. Built around a See-Think-Sketch-Act loop and adaptive token gating, the framework decomposes tasks into subgoals, renders scene-grounded sketches, and executes actions with a flow-matching controller, enabling reactive corrections and human supervision. A three-stage curriculum aligns language, sketches, and actions, supported by a rich dataset combining interleaved images, text, sketches, and trajectories; ablations and real-robot experiments demonstrate superior long-horizon performance, robustness to dynamics, and improved interpretability. The work advances practical grounding and controllable planning in VLA systems, offering a scalable path toward transparent, human-in-the-loop robotic manipulation in cluttered and evolving environments.

Abstract

Long-horizon robotic manipulation is increasingly important for real-world deployment, requiring spatial disambiguation in complex layouts and temporal resilience under dynamic interaction. However, existing end-to-end and hierarchical Vision-Language-Action (VLA) policies often rely on text-only cues while keeping plan intent latent, which undermines referential grounding in cluttered or underspecified scenes, impedes effective task decomposition of long-horizon goals with close-loop interaction, and limits causal explanation by obscuring the rationale behind action choices. To address these issues, we first introduce Visual Sketch, an implausible visual intermediate that renders points, boxes, arrows, and typed relations in the robot's current views to externalize spatial intent, connect language to scene geometry. Building on Visual Sketch, we present Action-Sketcher, a VLA framework that operates in a cyclic See-Think-Sketch-Act workflow coordinated by adaptive token-gated strategy for reasoning triggers, sketch revision, and action issuance, thereby supporting reactive corrections and human interaction while preserving real-time action prediction. To enable scalable training and evaluation, we curate diverse corpus with interleaved images, text, Visual Sketch supervision, and action sequences, and train Action-Sketcher with a multi-stage curriculum recipe that combines interleaved sequence alignment for modality unification, language-to-sketch consistency for precise linguistic grounding, and imitation learning augmented with sketch-to-action reinforcement for robustness. Extensive experiments on cluttered scenes and multi-object tasks, in simulation and on real-world tasks, show improved long-horizon success, stronger robustness to dynamic scene changes, and enhanced interpretability via editable sketches and step-wise plans. Project website: https://action-sketcher.github.io

Figures (13)

• Figure 1: Overview of Action-Sketcher. Our framework operates in a See-Think-Sketch-Act loop, where a foundation model first performs temporal and spatial reasoning to decompose a high-level instruction (e.g., "Clean the objects on the table") into a subtask and a corresponding Visual Sketch. This sketch, composed of primitives like points, boxes, and arrows, serves as an explicit, human-readable plan that guides a low-level policy to generate robust action sequences. This methodology enables three key capabilities: (bottom left) long-horizon planning through task decomposition, (bottom middle) explicit spatial reasoning by grounding instructions in scene geometry, and (bottom right) seamless human-in-the-loop adaptability via direct sketch correction and intent supervision.
• Figure 2: Overview of Action– Sketcher. The model runs an event–driven loop that (i) summarizes the next subtask, (ii) emits a compact Visual Sketch (points, boxes, arrows, relations) that externalizes spatial intent, and (iii) synthesizes an action chunk conditioned on that sketch and the robot state. The explicit intermediate supports targeted supervision, on–the–fly correction, and reliable long–horizon execution within a single–model architecture.
• Figure 3: Real-World Demonstrations of Action-Sketcher. Qualitative rollouts on long-horizon and spatial manipulation tasks. Our framework generates explicit Visual Sketches (overlaid points, boxes, and arrows) to ground high-level reasoning into low-level actions, successfully completing tasks like tidying a tabletop and pouring tea in cluttered environments.
• Figure 4: Failure analysis of Action-Sketcher. Most errors arise in the Reasoning Mode, primarily due to inaccurate Visual Sketch generation.
• Figure 5: The distribution of the entire training dataset.