From Observation to Action: Latent Action-based Primitive Segmentation for VLA Pre-training in Industrial Settings

TL;DR

The paper addresses the data bottleneck in industrialVision-Language-Action pretraining by proposing LAPS, an unsupervised pipeline that converts continuous industrial video into a structured library of action primitives. It combines a Motion Tokenizer to encode motion dynamics, a Latent Action Energy metric to detect semantic boundaries, and a two-stage clustering approach (frozen Transformer embeddings followed by cosine k-means) to discover a finite set of primitives, validated with a Vision-Language Model via ICSS. Experiments on public benchmarks and a real-world motor assembly dataset show that Latent Action Energy improves segmentation quality, and the resulting primitives exhibit high semantic coherence, demonstrating the method’s practicality for scalable VLA pre-training in manufacturing. The approach offers a scalable, automated data-sourcing pathway for embodied AI, with potential extensions to broader domains and future work to connect the latent space to executable control.

Abstract

We present a novel unsupervised framework to unlock vast unlabeled human demonstration data from continuous industrial video streams for Vision-Language-Action (VLA) model pre-training. Our method first trains a lightweight motion tokenizer to encode motion dynamics, then employs an unsupervised action segmenter leveraging a novel "Latent Action Energy" metric to discover and segment semantically coherent action primitives. The pipeline outputs both segmented video clips and their corresponding latent action sequences, providing structured data directly suitable for VLA pre-training. Evaluations on public benchmarks and a proprietary electric motor assembly dataset demonstrate effective segmentation of key tasks performed by humans at workstations. Further clustering and quantitative assessment via a Vision-Language Model confirm the semantic coherence of the discovered action primitives. To our knowledge, this is the first fully automated end-to-end system for extracting and organizing VLA pre-training data from unstructured industrial videos, offering a scalable solution for embodied AI integration in manufacturing.

Figures (5)

• Figure 1: Example of our segmentation approach using Latent Action Energy from a Motion Tokenizer. Action boundaries (red circles) correspond to transitions from high energy to baseline, indicating action completion. The pipeline outputs the Latent Action Sequence (bottom codes), providing structured representations for VLA pre-training.
• Figure 2: Overview of the LAPS pipeline: (1) Motion Tracking extracts motion keypoints from raw video using a point tracker. (2) Action Detection & Segmentation generates a latent vector stream via a motion tokenizer and identifies action boundaries to segment latent vectors, video clips, and action codes. (3) Semantic Action Clustering groups the segmented latent vectors into meaningful semantic action clusters.
• Figure 3: Sliding-window tokenization: A motion tokenizer converts the video stream into a sequence of discrete latent action indices $c_t \in \{0,\ldots,2047\}$, the main output for VLA pre-training. Action detection and clustering use the corresponding continuous quantized vectors.
• Figure 4: Qualitative comparison of our $E_{\text{action}}$ (blue) and Optical Flow (red) over 60 seconds. Our latent action energy shows clear, sustained peaks during actions and sharp drops at ground truth semantic boundaries (dashed lines), while optical flow is noisy and reflects only physical movement, not task phases.
• Figure 5: UMAP visualization of action primitive embeddings colored by k-means cluster ID. Distinct, well-separated clusters that correspond to real workstation tasks confirmed through manual inspection.