Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DISCOVER: A Data-driven Interactive System for Comprehensive Observation, Visualization, and ExploRation of Human Behaviour

Dominik Schiller, Tobias Hallmen, Daksitha Withanage Don, Elisabeth André, Tobias Baur

TL;DR

DISCOVER addresses the bottlenecks in analyzing human behavior by providing a modular, open‑source framework that democratizes access to advanced computational tools. The system integrates a NOVA-based UI, a central annotation database, shared media storage, a processing server, and an LLM-powered assistant to enable interactive semantic content exploration, visual inspection, aided annotation, and multimodal scene search. It combines state‑of‑the‑art face, voice, and multimodal feature extractors with a cooperative machine learning workflow, allowing researchers to train and refine models within the interface. Together, these components support scalable, collaborative analysis of multimodal behavioral data, reducing technical barriers and enabling rapid exploratory data analysis across disciplines.

Abstract

Understanding human behavior is a fundamental goal of social sciences, yet its analysis presents significant challenges. Conventional methodologies employed for the study of behavior, characterized by labor-intensive data collection processes and intricate analyses, frequently hinder comprehensive exploration due to their time and resource demands. In response to these challenges, computational models have proven to be promising tools that help researchers analyze large amounts of data by automatically identifying important behavioral indicators, such as social signals. However, the widespread adoption of such state-of-the-art computational models is impeded by their inherent complexity and the substantial computational resources necessary to run them, thereby constraining accessibility for researchers without technical expertise and adequate equipment. To address these barriers, we introduce DISCOVER -- a modular and flexible, yet user-friendly software framework specifically developed to streamline computational-driven data exploration for human behavior analysis. Our primary objective is to democratize access to advanced computational methodologies, thereby enabling researchers across disciplines to engage in detailed behavioral analysis without the need for extensive technical proficiency. In this paper, we demonstrate the capabilities of DISCOVER using four exemplary data exploration workflows that build on each other: Interactive Semantic Content Exploration, Visual Inspection, Aided Annotation, and Multimodal Scene Search. By illustrating these workflows, we aim to emphasize the versatility and accessibility of DISCOVER as a comprehensive framework and propose a set of blueprints that can serve as a general starting point for exploratory data analysis.

DISCOVER: A Data-driven Interactive System for Comprehensive Observation, Visualization, and ExploRation of Human Behaviour

TL;DR

DISCOVER addresses the bottlenecks in analyzing human behavior by providing a modular, open‑source framework that democratizes access to advanced computational tools. The system integrates a NOVA-based UI, a central annotation database, shared media storage, a processing server, and an LLM-powered assistant to enable interactive semantic content exploration, visual inspection, aided annotation, and multimodal scene search. It combines state‑of‑the‑art face, voice, and multimodal feature extractors with a cooperative machine learning workflow, allowing researchers to train and refine models within the interface. Together, these components support scalable, collaborative analysis of multimodal behavioral data, reducing technical barriers and enabling rapid exploratory data analysis across disciplines.

Abstract

Understanding human behavior is a fundamental goal of social sciences, yet its analysis presents significant challenges. Conventional methodologies employed for the study of behavior, characterized by labor-intensive data collection processes and intricate analyses, frequently hinder comprehensive exploration due to their time and resource demands. In response to these challenges, computational models have proven to be promising tools that help researchers analyze large amounts of data by automatically identifying important behavioral indicators, such as social signals. However, the widespread adoption of such state-of-the-art computational models is impeded by their inherent complexity and the substantial computational resources necessary to run them, thereby constraining accessibility for researchers without technical expertise and adequate equipment. To address these barriers, we introduce DISCOVER -- a modular and flexible, yet user-friendly software framework specifically developed to streamline computational-driven data exploration for human behavior analysis. Our primary objective is to democratize access to advanced computational methodologies, thereby enabling researchers across disciplines to engage in detailed behavioral analysis without the need for extensive technical proficiency. In this paper, we demonstrate the capabilities of DISCOVER using four exemplary data exploration workflows that build on each other: Interactive Semantic Content Exploration, Visual Inspection, Aided Annotation, and Multimodal Scene Search. By illustrating these workflows, we aim to emphasize the versatility and accessibility of DISCOVER as a comprehensive framework and propose a set of blueprints that can serve as a general starting point for exploratory data analysis.
Paper Structure (32 sections, 6 figures)

This paper contains 32 sections, 6 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Framework
  4. User Interface
  5. Annotation Database
  6. Media File Storage
  7. Processing Server
  8. Assistant
  9. Modules
  10. Face
  11. Bounding Box Detector
  12. Landmarks and Meshes
  13. Action Units
  14. Facial Expression
  15. Voice
  16. ...and 17 more sections

Figures (6)

  • Figure 1: Overview of the system components for the DISCOVER eco system
  • Figure 2: NOVA allows to visualise various media and signal types and supports different annotation schemes. From top downwards: upper-body videos along with face tracking, audio streams of two persons during an interaction, and activation of the action units. In the lower part, free-value, discrete, and continuous annotation tiers are displayed.
  • Figure 3: Schematic representation of the exploratory data analysis workflow. 1. Explore the content of a conversation using the DISCOVER assistant. 2. Compute and visualize behavioral cues. 3. Annotate additional indicators with the help of cooperative machine learning. 4. Identify scenes that consist of an interplay of several indicators. Every step in the workflow can be repeated and adapted as necessary.
  • Figure 4: A user engages with the assistant to explore the semantics of a conversation interactively.
  • Figure 5: Visual data inspection with NOVA and DISCOVER. The session overview provides a comprehensive summary of the extracted behavior indicators across the entire session. Zooming in at the bottom enables a detailed analysis of identified scenes of interest.
  • ...and 1 more figures