Meta-Control: Automatic Model-based Control Synthesis for Heterogeneous Robot Skills

TL;DR

Meta-Control addresses the challenge of diverse, conflicting real-world manipulation requirements by introducing an LLM-enabled automatic model-based control synthesis framework. It formulates skills as a composable hierarchical system with task-space dynamics $\\dot z = h(z,v)$ and tracking-space dynamics $\\dot x = f(x,u)$, and leverages templates to ground model/controller design while enabling task-specific customization. The approach uses a three-level pipeline—Strategy, Data Flow, and Parameter—driven by Socratic prompting and a code-extractor to generate executable control code, with reflection used to recover from errors. Empirically, Meta-Control demonstrates synthesis of challenging heterogeneous skills across simulation and real-robot experiments, with ablation showing benefits from both hierarchy and templates, and with model-based design providing robustness and generalization across embodiments and task attributes. This work advances autonomous robotic autonomy by reducing manual tuning, enabling explainable, verifiable control synthesis, and enabling safer, more adaptable manipulation in open-world settings.

Abstract

The requirements for real-world manipulation tasks are diverse and often conflicting; some tasks require precise motion while others require force compliance; some tasks require avoidance of certain regions, while others require convergence to certain states. Satisfying these varied requirements with a fixed state-action representation and control strategy is challenging, impeding the development of a universal robotic foundation model. In this work, we propose Meta-Control, the first LLM-enabled automatic control synthesis approach that creates customized state representations and control strategies tailored to specific tasks. Our core insight is that a meta-control system can be built to automate the thought process that human experts use to design control systems. Specifically, human experts heavily use a model-based, hierarchical (from abstract to concrete) thought model, then compose various dynamic models and controllers together to form a control system. Meta-Control mimics the thought model and harnesses LLM's extensive control knowledge with Socrates' "art of midwifery" to automate the thought process. Meta-Control stands out for its fully model-based nature, allowing rigorous analysis, generalizability, robustness, efficient parameter tuning, and reliable real-time execution.

Figures (8)

• Figure 1: Opening the door, moving safely, and balancing are heterogeneous skills and have conflicting requirements that can be difficult to satisfy with a fixed control strategy and fixed state-action representation. Meta-Control addresses the challenge with a composable hierarchical control formulation and LLM, enabling automatic model-based control synthesis. The synthesized skill has customized state action representations, dynamic models, and controllers that perfectly align with the objectives and requirements.
• Figure 2: Comparison between Meta-Control and a trajectory planning-based method on a real robot for wiping the board and opening the cabinet. The trajectory-based method fails to erase the mark because it neglects force requirements. Opening the cabinet with a trajectory-based controller leads to cabinet displacement because the planned trajectory does not precisely align with the door's swing path, which may damage the door if the cabinet is fixed. In contrast, Meta-Control addressed these challenges with properly customized control systems.
• Figure 3: Overview of Meta-Control: The user only needs to provide a skill description. Meta-Control then leverages the control knowledge of LLMs to synthesize skills through a three-level pipeline: strategy level, data flow level, and parameter level. For each level, we have designed a generic prompt with placeholders, which are dynamically replaced with user input or code extracted from the LLM response during runtime, utilizing a code extractor to make the prompt task-specific. The extracted code is also used to construct the control system. At each level, if the LLM-generated code results in an error, a reflection phase is initiated. We have embedded design principles and checklists of common errors within the design and reflection prompts to assist the LLM in producing correct code. The generic prompt design with placeholders allows Meta-Control to generalize to unseen tasks without modification.
• Figure 4: Five manipulation tasks that have inherently different challenges and requirements. For example, the balance task requires an accurate and high-frequency feedback controller. The safe pick and place task requires guaranteeing collision avoidance for the whole robot arm. The open door task requires properly handling articulated objects; and the executed trajectory has to perfectly match the swing path.
• Figure 5: Meta-Control can automatically identify hyper-parameters that require tuning and tune them to accomplish challenging tasks. The figure shows the trajectory of the arm-held cart-pole system before and after tuning the synthesized controller. The hyper-parameters $Q=$diag$(10, 1, 100, 1)$, $R=0.01$ are chosen and tuned by the LLM with only 2 rounds of trial-and-error.