Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bridging Scale Discrepancies in Robotic Control via Language-Based Action Representations

Yuchi Zhang, Churui Sun, Shiqi Liang, Diyuan Liu, Chao Ji, Wei-Nan Zhang, Ting Liu

TL;DR

This paper tackles distribution shifts in robotic action data by introducing a language-based, semantically grounded motion representation to normalize actions across diverse robots and tasks. It pairs an action tokenizer with an adaptive, multi-scale motion generation module and a two-stage training regime to align motion with language tokens, enabling robust pretraining and transfer to downstream manipulation benchmarks. Through extensive experiments on LIBERO and SimplerEnv, the approach demonstrates improved generalization and transferability, outperforming several state-of-the-art baselines. The results suggest that abstracting actions into directional motion language reduces modality gaps and accelerates learning in language-conditioned robotic manipulation.

Abstract

Recent end-to-end robotic manipulation research increasingly adopts architectures inspired by large language models to enable robust manipulation. However, a critical challenge arises from severe distribution shifts between robotic action data, primarily due to substantial numerical variations in action commands across diverse robotic platforms and tasks, hindering the effective transfer of pretrained knowledge. To address this limitation, we propose a semantically grounded linguistic representation to normalize actions for efficient pretraining. Unlike conventional discretized action representations that are sensitive to numerical scales, the motion representation specifically disregards numeric scale effects, emphasizing directionality instead. This abstraction mitigates distribution shifts, yielding a more generalizable pretraining representation. Moreover, using the motion representation narrows the feature distance between action tokens and standard vocabulary tokens, mitigating modality gaps. Multi-task experiments on two benchmarks demonstrate that the proposed method significantly improves generalization performance and transferability in robotic manipulation tasks.

Bridging Scale Discrepancies in Robotic Control via Language-Based Action Representations

TL;DR

This paper tackles distribution shifts in robotic action data by introducing a language-based, semantically grounded motion representation to normalize actions across diverse robots and tasks. It pairs an action tokenizer with an adaptive, multi-scale motion generation module and a two-stage training regime to align motion with language tokens, enabling robust pretraining and transfer to downstream manipulation benchmarks. Through extensive experiments on LIBERO and SimplerEnv, the approach demonstrates improved generalization and transferability, outperforming several state-of-the-art baselines. The results suggest that abstracting actions into directional motion language reduces modality gaps and accelerates learning in language-conditioned robotic manipulation.

Abstract

Recent end-to-end robotic manipulation research increasingly adopts architectures inspired by large language models to enable robust manipulation. However, a critical challenge arises from severe distribution shifts between robotic action data, primarily due to substantial numerical variations in action commands across diverse robotic platforms and tasks, hindering the effective transfer of pretrained knowledge. To address this limitation, we propose a semantically grounded linguistic representation to normalize actions for efficient pretraining. Unlike conventional discretized action representations that are sensitive to numerical scales, the motion representation specifically disregards numeric scale effects, emphasizing directionality instead. This abstraction mitigates distribution shifts, yielding a more generalizable pretraining representation. Moreover, using the motion representation narrows the feature distance between action tokens and standard vocabulary tokens, mitigating modality gaps. Multi-task experiments on two benchmarks demonstrate that the proposed method significantly improves generalization performance and transferability in robotic manipulation tasks.

Paper Structure

This paper contains 24 sections, 6 equations, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. End-to-End Action Generation
  4. Action Generation Assisted by Textual Guidance
  5. Methods
  6. Action Tokenizer
  7. Motion Generation
  8. Motion Representation
  9. Threshold
  10. Window
  11. Design Justification
  12. Two-Stage Training
  13. Experimental Setup
  14. Research Questions
  15. Baseline Methods
  16. ...and 9 more sections

Figures (3)

  • Figure 1: The proposed motion data generation pipeline. The left part illustrates the distributions of specific execution actions across different types of datasets; The middle part presents our threshold- and window-based detection framework along with its proposed improvements; The right part depicts the structure and representation of the generated motion outputs.
  • Figure 2: Two-stage training on Qwen2.5 (0.5B, 1.5B, 3B): pretraining predicts motion tokens; fine-tuning predicts motion then action tokens. Image tokens denote the observed visual input; text tokens denote the task instruction; motion tokens denote our proposed motion language; and action tokens denote the discrete action representation.
  • Figure 3: Comparison of four different experimental setups