Investigating the Effectiveness of a Socratic Chain-of-Thoughts Reasoning Method for Task Planning in Robotics, A Case Study

TL;DR

This work investigates whether Socratic Chain-of-Thoughts (SocraCoT) can enhance zero-shot task planning for robotics by combining multi-agent, Socratic reasoning with chain-of-thought prompts to generate subtasks and robotic control code. Using a Webots-simulated Tiago robot and GPT-4(Omni), the study compares Non-CoT, CoT, and SocraCoT strategies on an object-search task across twenty runs, measuring task success and execution time. Results suggest SocraCoT improves the logical quality of subtasks and reduces coding errors relative to baseline CoT, albeit with higher compute and latency costs; limitations include perception challenges and potential prompt-driven variability. The authors propose EVINCE-LoC as an enhanced framework to boost performance in complex or dynamic scenarios, outlining avenues for prompt engineering and integration with localization-perception modules. Overall, the paper contributes a first demonstration of combining Socratic reasoning with CoT for robotic task planning and outlines practical steps to scale this approach in real-world-like settings.

Abstract

Large language models (LLMs) have demonstrated unprecedented capability in reasoning with natural language. Coupled with this development is the emergence of embodied AI in robotics. Despite showing promise for verbal and written reasoning tasks, it remains unknown whether LLMs are capable of navigating complex spatial tasks with physical actions in the real world. To this end, it is of interest to investigate applying LLMs to robotics in zero-shot learning scenarios, and in the absence of fine-tuning - a feat which could significantly improve human-robot interaction, alleviate compute cost, and eliminate low-level programming tasks associated with robot tasks. To explore this question, we apply GPT-4(Omni) with a simulated Tiago robot in Webots engine for an object search task. We evaluate the effectiveness of three reasoning strategies based on Chain-of-Thought (CoT) sub-task list generation with the Socratic method (SocraCoT) (in order of increasing rigor): (1) Non-CoT/Non-SocraCoT, (2) CoT only, and (3) SocraCoT. Performance was measured in terms of the proportion of tasks successfully completed and execution time (N = 20). Our preliminary results show that when combined with chain-of-thought reasoning, the Socratic method can be used for code generation for robotic tasks that require spatial awareness. In extension of this finding, we propose EVINCE-LoC; a modified EVINCE method that could further enhance performance in highly complex and or dynamic testing scenarios.

Figures (8)

• Figure 1: Experimental setup for evaluating the effectiveness of a Socratic CoT reasoning strategy when applied to robotics in a simulated environment. Leftmost picture shows the simulated Tiago robot, which is equipped with an odometer placed on its chassis, two differential wheels and one Astra Depth camera placed on its head, which can output both RGB and depth images. The robot also has a Hokuyo Lidar on its chassis for obstacle avoidance. The middle picture details the objects used for testing and the possible actions for the robot. The rightmost picture shows the 3 scenarios we tested and benchmarked against, the first one being vanilla setup without any chain-of-thought nor Socratic participation, the second one includes chain-of-thought strategy, and the third one employs Socratic method on both chain-of-thought strategy and task code generation for completing the designated task described in natural language.
• Figure 2: Proposed SocraCoT framework in high-level view, it consists of an actor LLM and an examiner LLM, whose roles get interchanged as they are "debating" over many iterations. The bottom component is responsible for constructing prompt, making queries to LLM API endpoint, collecting responses, and postprocessing the responses to control the robot. This module also subscribes to the sensor data published by the Webots ROS2 bridge, and makes decisions by calling the "move-robot" function generated by the LLM. The upper right component is responsible for LLM query and debate, by firstly asking the LLM (actor) to generate a list of subtasks, then by employing Socratic method, makes another LLM (critic) query to generate a revised list of subtasks. Afterwards, queries are made to LLM (moderator) for generating the code for the incomplete function "move-robot". The code generated is then reviewed and improved to ensure its robustness. The generated code is then plugged into the running Python script and executed in real time.
• Figure 3: Prompt template used for breaking down the high-level task.
• Figure 4: Prompt template used for generating the code for completing the robot controller function.
• Figure 5: Prompt template used for revising the code through the method of counterfactual reasoning.