A Backbone for Long-Horizon Robot Task Understanding

TL;DR

Long-horizon robot tasks suffer from poor generalization and data inefficiency in end-to-end learning. This work introduces the Therblig-Based Backbone Framework (TBBF), a structured backbone that decomposes tasks into therbligs and integrates an offline segmentation network (MGSF), an action-registration module (ActionREG), and a LLM-alignment visual-correction policy (LAP-VC) to enable one-shot transfer to new scenarios. Empirical results show high therblig segmentation recall (94.37%), and robust online task success (94.4% in simple scenarios, 80% in complex scenarios), with LAP-VC achieving strong alignment. The framework improves interpretability, data efficiency, and generalization, enabling more reliable long-horizon robot manipulation in cluttered and dynamic environments. The authors also outline future work on larger datasets, 3D configurations, and deploying a local LLM to reduce latency.

Abstract

End-to-end robot learning, particularly for long-horizon tasks, often results in unpredictable outcomes and poor generalization. To address these challenges, we propose a novel Therblig-Based Backbone Framework (TBBF) as a fundamental structure to enhance interpretability, data efficiency, and generalization in robotic systems. TBBF utilizes expert demonstrations to enable therblig-level task decomposition, facilitate efficient action-object mapping, and generate adaptive trajectories for new scenarios. The approach consists of two stages: offline training and online testing. During the offline training stage, we developed the Meta-RGate SynerFusion (MGSF) network for accurate therblig segmentation across various tasks. In the online testing stage, after a one-shot demonstration of a new task is collected, our MGSF network extracts high-level knowledge, which is then encoded into the image using Action Registration (ActionREG). Additionally, Large Language Model (LLM)-Alignment Policy for Visual Correction (LAP-VC) is employed to ensure precise action registration, facilitating trajectory transfer in novel robot scenarios. Experimental results validate these methods, achieving 94.37% recall in therblig segmentation and success rates of 94.4% and 80% in real-world online robot testing for simple and complex scenarios, respectively. Supplementary material is available at: https://sites.google.com/view/therbligsbasedbackbone/home

Figures (8)

• Figure 1: Concept of the Proposed Robot Task Understanding System: extracts key backbone of complex tasks and uses context from a single demonstration to understand relevant objects and actions.
• Figure 2: Detailed Decomposition of a Robotic Task into therbligs. The sequence containing: Rest (R), Transport Empty (TE), Delay (D), Grasp (G), Transport Load (TL), Use (U) and Release (R).
• Figure 3: Overview of the proposed TBBF. This pipeline integrates offline training and online testing stages. During offline training, human experts provide demonstrations and label robot trajectories into therbligs, which are then used to train the MGSF network. In the online testing stage, the trained MGSF network segments new tasks into Therblig-level actions. ActionREG registers these actions into new configurations, and LAP-VC is utilized for error compensation. Finally, YOLOv8 and PCA are used to match new configurations. Arrows indicate the starting and ending points of the trajectory flow.
• Figure 4: Detailed architecture of the MGSF network. The MGSF network integrates BiLSTM and Transformer sub-networks to capture sequential dependencies and use a meta-recursive gated fusion mechanism to dynamically combine the outputs of these sub-networks.
• Figure 5: Details of the action registration, context matching, and new trajectory generating process. Arrows indicate the direction of trajectory.