Social Behaviour Understanding using Deep Neural Networks: Development of Social Intelligence Systems
Ethan Lim Ding Feng, Zhi-Wei Neo, Aaron William De Silva, Kellie Sim, Hong-Ray Tan, Thi-Thanh Nguyen, Karen Wei Ling Koh, Wenru Wang, Hoang D. Nguyen
TL;DR
The paper addresses transforming multimodal social data into actionable social intelligence by modeling social signals and behaviours with deep neural networks. It proposes a five-component framework—Information Fusion, Person and Object Detection, Social Signal Understanding, Behavioural Understanding, and Context Understanding—and discusses cross-modality transformers for multimodal integration. Through three health-oriented applications (depression detection from voice, activity recognition for fall prevention, and cognitive impairment screening from handwriting and movement), it demonstrates real-time feasibility and potential clinical impact. Overall, the work advances social computing and mobile health informatics by providing a concrete design and deployment path for socially intelligent agents.
Abstract
With the rapid development in artificial intelligence, social computing has evolved beyond social informatics toward the birth of social intelligence systems. This paper, therefore, takes initiatives to propose a social behaviour understanding framework with the use of deep neural networks for social and behavioural analysis. The integration of information fusion, person and object detection, social signal understanding, behaviour understanding, and context understanding plays a harmonious role to elicit social behaviours. Three systems, including depression detection, activity recognition and cognitive impairment screening, are developed to evidently demonstrate the importance of social intelligence. The study considerably contributes to the cumulative development of social computing and health informatics. It also provides a number of implications for academic bodies, healthcare practitioners, and developers of socially intelligent agents.