Action Contextualization: Adaptive Task Planning and Action Tuning using Large Language Models

TL;DR

A novel framework to achieve action contextualization, aimed at tailoring robot actions to the context of specific tasks, thereby enhancing adaptability through applying LLM-derived contextual insights is introduced.

Abstract

Large Language Models (LLMs) present a promising frontier in robotic task planning by leveraging extensive human knowledge. Nevertheless, the current literature often overlooks the critical aspects of robots' adaptability and error correction. This work aims to overcome this limitation by enabling robots to modify their motions and select the most suitable task plans based on the context. We introduce a novel framework to achieve action contextualization, aimed at tailoring robot actions to the context of specific tasks, thereby enhancing adaptability through applying LLM-derived contextual insights. Our framework integrates motion metrics that evaluate robot performances for each motion to resolve redundancy in planning. Moreover, it supports online feedback between the robot and the LLM, enabling immediate modifications to the task plans and corrections of errors. An overall success rate of 81.25% has been achieved through extensive experimental validation. Finally, when integrated with dynamical system (DS)-based robot controllers, the robotic arm-hand system demonstrates its proficiency in autonomously executing LLM-generated motion plans for sequential table-clearing tasks, rectifying errors without human intervention, and showcasing robustness against external disturbances. Our proposed framework also features the potential to be integrated with modular control approaches, significantly enhancing robots' adaptability and autonomy in performing sequential tasks in the real world.

Figures (12)

• Figure 1: The proposed framework for performing context-aware task planning and action parameter tuning for robotic sequential tasks. The red arrow lines indicate the retune or replan of the parameters, while the grey lines indicate the process for the real robotic system to execute the task once the action plan is tuned
• Figure 2: Instances of actions from task plan and the corresponding checks from evaluation plan. Observations from the checks provide useful inference to LLM: (a) points to safe, collision-free and timely execution of approach, resulting in the robot being close enough to grasp the cup; (b) the action failed due to a false check_motion_health(), which can be overcome by tuning the motion by using a lower speed value; (c) the action was successful and the cup is now on the kitchen sink
• Figure 3: The input scene and description generated by LLM, accompanied by the extracted objects and locations
• Figure 4: Illustration of the motion re-tune process and the associated LLM's reasoning process using the task of clearing a paper ball. The execution result in the simulation is used as feedback to the LLM for retuning
• Figure 5: Motion scores after retuning and replanning. P0 and P1 indicate the original task plan and first re-plan, and R1, R2, and R3 indicate the retune of motions in each task plan, respectively. The shaded area in red color indicates failures