Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Understanding of Human-Robot Social Interactions through Multimodal Distillation

Tongfei Bian, Mathieu Chollet, Tanaya Guha

TL;DR

The paper tackles robust, real-time social understanding in egocentric human–robot interactions under partial and noisy inputs. It introduces a two-stage knowledge distillation framework in which a multimodal teacher (SocialC3D) trains a lightweight body-pose–only student (SocialEgoMobile) using an InfoNCE loss and input corruption to preserve rich social knowledge. Empirical results on JPL-Social and HARPER show that the student achieves substantial accuracy gains and maintains robustness with up to 51% input corruption, while using only about 1% of the teacher’s parameters and roughly 12% of its latency. This approach enables practical deployment of robust social understanding on resource-constrained robots, advancing real-world HRI capabilities.

Abstract

There is a growing need for social robots and intelligent agents that can effectively interact with and support users. For the interactions to be seamless, the agents need to analyse social scenes and behavioural cues from their (robot's) perspective. Works that model human-agent interactions in social situations are few; and even those existing ones are computationally too intensive to be deployed in real time or perform poorly in real-world scenarios when only limited information is available. We propose a knowledge distillation framework that models social interactions through various multimodal cues, and yet is robust against incomplete and noisy information during inference. We train a teacher model with multimodal input (body, face and hand gestures, gaze, raw images) that transfers knowledge to a student model which relies solely on body pose. Extensive experiments on two publicly available human-robot interaction datasets demonstrate that our student model achieves an average accuracy gain of 14.75% over competitive baselines on multiple downstream social understanding tasks, even with up to 51% of its input being corrupted. The student model is also highly efficient - less than 1% in size of the teacher model in terms of parameters and its latency is 11.9% of the teacher model. Our code and related data are available at github.com/biantongfei/SocialEgoMobile.

Robust Understanding of Human-Robot Social Interactions through Multimodal Distillation

TL;DR

The paper tackles robust, real-time social understanding in egocentric human–robot interactions under partial and noisy inputs. It introduces a two-stage knowledge distillation framework in which a multimodal teacher (SocialC3D) trains a lightweight body-pose–only student (SocialEgoMobile) using an InfoNCE loss and input corruption to preserve rich social knowledge. Empirical results on JPL-Social and HARPER show that the student achieves substantial accuracy gains and maintains robustness with up to 51% input corruption, while using only about 1% of the teacher’s parameters and roughly 12% of its latency. This approach enables practical deployment of robust social understanding on resource-constrained robots, advancing real-world HRI capabilities.

Abstract

There is a growing need for social robots and intelligent agents that can effectively interact with and support users. For the interactions to be seamless, the agents need to analyse social scenes and behavioural cues from their (robot's) perspective. Works that model human-agent interactions in social situations are few; and even those existing ones are computationally too intensive to be deployed in real time or perform poorly in real-world scenarios when only limited information is available. We propose a knowledge distillation framework that models social interactions through various multimodal cues, and yet is robust against incomplete and noisy information during inference. We train a teacher model with multimodal input (body, face and hand gestures, gaze, raw images) that transfers knowledge to a student model which relies solely on body pose. Extensive experiments on two publicly available human-robot interaction datasets demonstrate that our student model achieves an average accuracy gain of 14.75% over competitive baselines on multiple downstream social understanding tasks, even with up to 51% of its input being corrupted. The student model is also highly efficient - less than 1% in size of the teacher model in terms of parameters and its latency is 11.9% of the teacher model. Our code and related data are available at github.com/biantongfei/SocialEgoMobile.
Paper Structure (18 sections, 6 equations, 4 figures, 3 tables)

This paper contains 18 sections, 6 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Social Intent Prediction
  4. Non-Verbal Cues in Multimodal HRI
  5. Methodology
  6. Problem Definition
  7. Teacher Model
  8. Student Model
  9. Knowledge Distillation
  10. Input Corruption
  11. Experiments
  12. Data
  13. Experiment Setting
  14. Performance of the Teacher Model
  15. Robustness Analysis on the Student Model
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Our knowledge distillation framework uses SocialC3D as the teacher model, which fuses raw images, body, face, hand gestures, and gaze information, producing a multimodal social representation. The lightweight student model, SocialEgoMobile, uses only body pose features to output social representations. The framework maximises the mutual information of the social representations of the teacher and student model, to transfer social knowledge.
  • Figure 2: We set the FPS of two datasets to 10 and limited the observation window to the first second of social interactions.
  • Figure 3: Knowledge distillation (KD) consistently improves the performance of the student model, SocialEgoMobile, under spatio-temporal corruption on all three downstream tasks, interaction intent, attitude, and social action forecast. Improvements on downstream task accuracy through distillation are labelled.
  • Figure 4: Comparison of SocialC3D using different input modalities and the its impact on SocialEgoMobile (SEM). Accuracy drops due to missing modalities are labelled.