Toward Grounded Commonsense Reasoning

TL;DR

This work proposes an approach that leverages an LLM and vision language model (VLM) to help a robot actively perceive its environment to perform grounded commonsense reasoning and finds an average 12.9% improvement on the MessySurfaces benchmark and an average 15% improvement on the robot experiments over baselines that do not use active perception.

Abstract

Consider a robot tasked with tidying a desk with a meticulously constructed Lego sports car. A human may recognize that it is not appropriate to disassemble the sports car and put it away as part of the "tidying." How can a robot reach that conclusion? Although large language models (LLMs) have recently been used to enable commonsense reasoning, grounding this reasoning in the real world has been challenging. To reason in the real world, robots must go beyond passively querying LLMs and actively gather information from the environment that is required to make the right decision. For instance, after detecting that there is an occluded car, the robot may need to actively perceive the car to know whether it is an advanced model car made out of Legos or a toy car built by a toddler. We propose an approach that leverages an LLM and vision language model (VLM) to help a robot actively perceive its environment to perform grounded commonsense reasoning. To evaluate our framework at scale, we release the MessySurfaces dataset which contains images of 70 real-world surfaces that need to be cleaned. We additionally illustrate our approach with a robot on 2 carefully designed surfaces. We find an average 12.9% improvement on the MessySurfaces benchmark and an average 15% improvement on the robot experiments over baselines that do not use active perception. The dataset, code, and videos of our approach can be found at https://minaek.github.io/grounded_commonsense_reasoning.

Figures (14)

• Figure 1: Parts 1 and 2 of Survey Interface.
• Figure 2: Parts 3 and 4 of Survey Interface.
• Figure 3: Real-World Commonsense Reasoning. We outline the steps of our framework with a robot. Notably, the LLM generates questions and "angles" for the arm to servo to (e.g., right of the banana). We also use the LLM to generate an action plan for each object -- each plan is converted to a sequence of skill primitives that are then executed by the robot.
• Figure 4: Code as Policies Interface for Real-Robot Execution. We define a simple programmatic interface for specifying robot skill primitives on in an object-oriented fashion. The interface is stateful; for robot primitives such as cleanup() and relocate(), the robot sets a designated receptacle via the special function set_designated(). On the right, we provide the actual execution trace produced by the LLM for the Kitchen Cleanup Desk (see \ref{['fig:kitchen_cleanup_desk']}).
• Figure 5: Real Robot Benchmark Accuracy. We construct benchmark questions for objects used with the real robot in similar manner to Section 4 in the main paper. Across both types of VLMs, our Ours-LLM beats Baseline Questions by an average of $13.5\%$, beats No Active Perception by an average of $18\%$, and beats No Questions by an average of $13.5\%$.