Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Loose Social-Interaction Recognition in Real-world Therapy Scenarios

Abid Ali, Rui Dai, Ashish Marisetty, Guillaume Astruc, Monique Thonnat, Jean-Marc Odobez, Susanne Thümmler, Francois Bremond

TL;DR

This work proposes a novel dual-path architecture to capture the loose interaction between two individuals and evaluates its model on real-world autism diagnoses such as the authors' Loose-Interaction dataset, and the publicly available Autism dataset for loose interactions.

Abstract

The computer vision community has explored dyadic interactions for atomic actions such as pushing, carrying-object, etc. However, with the advancement in deep learning models, there is a need to explore more complex dyadic situations such as loose interactions. These are interactions where two people perform certain atomic activities to complete a global action irrespective of temporal synchronisation and physical engagement, like cooking-together for example. Analysing these types of dyadic-interactions has several useful applications in the medical domain for social-skills development and mental health diagnosis. To achieve this, we propose a novel dual-path architecture to capture the loose interaction between two individuals. Our model learns global abstract features from each stream via a CNNs backbone and fuses them using a new Global-Layer-Attention module based on a cross-attention strategy. We evaluate our model on real-world autism diagnoses such as our Loose-Interaction dataset, and the publicly available Autism dataset for loose interactions. Our network achieves baseline results on the Loose-Interaction and SOTA results on the Autism datasets. Moreover, we study different social interactions by experimenting on a publicly available dataset i.e. NTU-RGB+D (interactive classes from both NTU-60 and NTU-120). We have found that different interactions require different network designs. We also compare a slightly different version of our method by incorporating time information to address tight interactions achieving SOTA results.

Loose Social-Interaction Recognition in Real-world Therapy Scenarios

TL;DR

This work proposes a novel dual-path architecture to capture the loose interaction between two individuals and evaluates its model on real-world autism diagnoses such as the authors' Loose-Interaction dataset, and the publicly available Autism dataset for loose interactions.

Abstract

The computer vision community has explored dyadic interactions for atomic actions such as pushing, carrying-object, etc. However, with the advancement in deep learning models, there is a need to explore more complex dyadic situations such as loose interactions. These are interactions where two people perform certain atomic activities to complete a global action irrespective of temporal synchronisation and physical engagement, like cooking-together for example. Analysing these types of dyadic-interactions has several useful applications in the medical domain for social-skills development and mental health diagnosis. To achieve this, we propose a novel dual-path architecture to capture the loose interaction between two individuals. Our model learns global abstract features from each stream via a CNNs backbone and fuses them using a new Global-Layer-Attention module based on a cross-attention strategy. We evaluate our model on real-world autism diagnoses such as our Loose-Interaction dataset, and the publicly available Autism dataset for loose interactions. Our network achieves baseline results on the Loose-Interaction and SOTA results on the Autism datasets. Moreover, we study different social interactions by experimenting on a publicly available dataset i.e. NTU-RGB+D (interactive classes from both NTU-60 and NTU-120). We have found that different interactions require different network designs. We also compare a slightly different version of our method by incorporating time information to address tight interactions achieving SOTA results.
Paper Structure (18 sections, 6 equations, 2 figures, 10 tables)

This paper contains 18 sections, 6 equations, 2 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Method
  4. Terminology description
  5. Convolution Backbone
  6. Abstract Projection
  7. Global-Layer-Attention
  8. Classification Head
  9. Temporal Synchronisation Modelling
  10. Experiments
  11. Loose-Interaction Dataset
  12. Public Datasets
  13. Experimental Details
  14. Experiments on ASD datasets
  15. Experiments on tight interactions
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Different dyadic interaction types.
  • Figure 2: Our proposed architecture consists of (a) the Convolution backbone, (b) the Abstract Projection module, and (c) the GLA module. The model takes the input (leader and assistant) and outputs the action prediction score through the classification head.