Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GRACE: Generating Socially Appropriate Robot Actions Leveraging LLMs and Human Explanations

Fethiye Irmak Dogan, Umut Ozyurt, Gizem Cinar, Hatice Gunes

TL;DR

GRACE addresses the challenge of generating socially appropriate robot actions by combining large language model (LLM) reasoning with human explanations. It first separates scenes into certain and uncertain using an uncertainty classifier, then uses LLMs for action appropriateness in certain cases, and employs a conditional autoencoder to refine predictions and generate explanations for uncertain cases. The approach leverages MannersDB and MannersDB+ to demonstrate that integrating human explanations improves accuracy and interpretability, outperforming baselines across multiple metrics. This bidirectional, explanation-aware framework has practical implications for trusted human-robot interaction and personalized robot behavior in social settings.

Abstract

When operating in human environments, robots need to handle complex tasks while both adhering to social norms and accommodating individual preferences. For instance, based on common sense knowledge, a household robot can predict that it should avoid vacuuming during a social gathering, but it may still be uncertain whether it should vacuum before or after having guests. In such cases, integrating common-sense knowledge with human preferences, often conveyed through human explanations, is fundamental yet a challenge for existing systems. In this paper, we introduce GRACE, a novel approach addressing this while generating socially appropriate robot actions. GRACE leverages common sense knowledge from LLMs, and it integrates this knowledge with human explanations through a generative network. The bidirectional structure of GRACE enables robots to refine and enhance LLM predictions by utilizing human explanations and makes robots capable of generating such explanations for human-specified actions. Our evaluations show that integrating human explanations boosts GRACE's performance, where it outperforms several baselines and provides sensible explanations.

GRACE: Generating Socially Appropriate Robot Actions Leveraging LLMs and Human Explanations

TL;DR

GRACE addresses the challenge of generating socially appropriate robot actions by combining large language model (LLM) reasoning with human explanations. It first separates scenes into certain and uncertain using an uncertainty classifier, then uses LLMs for action appropriateness in certain cases, and employs a conditional autoencoder to refine predictions and generate explanations for uncertain cases. The approach leverages MannersDB and MannersDB+ to demonstrate that integrating human explanations improves accuracy and interpretability, outperforming baselines across multiple metrics. This bidirectional, explanation-aware framework has practical implications for trusted human-robot interaction and personalized robot behavior in social settings.

Abstract

When operating in human environments, robots need to handle complex tasks while both adhering to social norms and accommodating individual preferences. For instance, based on common sense knowledge, a household robot can predict that it should avoid vacuuming during a social gathering, but it may still be uncertain whether it should vacuum before or after having guests. In such cases, integrating common-sense knowledge with human preferences, often conveyed through human explanations, is fundamental yet a challenge for existing systems. In this paper, we introduce GRACE, a novel approach addressing this while generating socially appropriate robot actions. GRACE leverages common sense knowledge from LLMs, and it integrates this knowledge with human explanations through a generative network. The bidirectional structure of GRACE enables robots to refine and enhance LLM predictions by utilizing human explanations and makes robots capable of generating such explanations for human-specified actions. Our evaluations show that integrating human explanations boosts GRACE's performance, where it outperforms several baselines and provides sensible explanations.
Paper Structure (19 sections, 4 equations, 3 figures, 3 tables)

This paper contains 19 sections, 4 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Datasets and Labels
  4. MannersDB and MannerDB+
  5. Categorization and Labels of Human Explanations
  6. METHODOLOGY
  7. Scene Clustering and Uncertainty Classification
  8. Action Appropriateness using LLMs
  9. Leveraging Explanations for Action Appropriateness
  10. Experiments and Results
  11. Implementation Details, Baselines and Metrics
  12. Uncertainty Classification
  13. LLM Predictions
  14. Leveraging Explanations for Action Appropriateness
  15. Results
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Flowchart of the proposed GRACE system.
  • Figure 2: The network structure of the GRACE autoencoder.
  • Figure 3: Given the human scores, the most likely explanations generated by the robot (prob. in parenthesis). The actions are vacuum cleaning, mopping the floor, carrying warm food, carrying cold food, carrying drinks, carrying small objects, carrying large objects, cleaning, and starting a conversation.