Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TCCT-Net: Two-Stream Network Architecture for Fast and Efficient Engagement Estimation via Behavioral Feature Signals

Alexander Vedernikov, Puneet Kumar, Haoyu Chen, Tapio Seppanen, Xiaobai Li

TL;DR

TCCT-Net addresses real-time engagement estimation on resource-limited devices by fusing temporal-spatial and temporal-frequency features derived from behavioral signals. It introduces a two-stream architecture: a Conformer-based temporal-spatial stream and a Continuous Wavelet Transform–based temporal-frequency stream, with decision-level fusion and a combined loss to optimize performance. The approach achieves competitive or superior accuracy with substantially fewer input signals and demonstrates order-of-magnitude speedups over image-based SOTA methods, validating its suitability for on-device deployment. The work also provides a principled data augmentation strategy and ablation analysis to isolate the contributions of each component, reinforcing the practicality of lightweight, signal-based engagement analysis.

Abstract

Engagement analysis finds various applications in healthcare, education, advertisement, services. Deep Neural Networks, used for analysis, possess complex architecture and need large amounts of input data, computational power, inference time. These constraints challenge embedding systems into devices for real-time use. To address these limitations, we present a novel two-stream feature fusion "Tensor-Convolution and Convolution-Transformer Network" (TCCT-Net) architecture. To better learn the meaningful patterns in the temporal-spatial domain, we design a "CT" stream that integrates a hybrid convolutional-transformer. In parallel, to efficiently extract rich patterns from the temporal-frequency domain and boost processing speed, we introduce a "TC" stream that uses Continuous Wavelet Transform (CWT) to represent information in a 2D tensor form. Evaluated on the EngageNet dataset, the proposed method outperforms existing baselines, utilizing only two behavioral features (head pose rotations) compared to the 98 used in baseline models. Furthermore, comparative analysis shows TCCT-Net's architecture offers an order-of-magnitude improvement in inference speed compared to state-of-the-art image-based Recurrent Neural Network (RNN) methods. The code will be released at https://github.com/vedernikovphoto/TCCT_Net.

TCCT-Net: Two-Stream Network Architecture for Fast and Efficient Engagement Estimation via Behavioral Feature Signals

TL;DR

TCCT-Net addresses real-time engagement estimation on resource-limited devices by fusing temporal-spatial and temporal-frequency features derived from behavioral signals. It introduces a two-stream architecture: a Conformer-based temporal-spatial stream and a Continuous Wavelet Transform–based temporal-frequency stream, with decision-level fusion and a combined loss to optimize performance. The approach achieves competitive or superior accuracy with substantially fewer input signals and demonstrates order-of-magnitude speedups over image-based SOTA methods, validating its suitability for on-device deployment. The work also provides a principled data augmentation strategy and ablation analysis to isolate the contributions of each component, reinforcing the practicality of lightweight, signal-based engagement analysis.

Abstract

Engagement analysis finds various applications in healthcare, education, advertisement, services. Deep Neural Networks, used for analysis, possess complex architecture and need large amounts of input data, computational power, inference time. These constraints challenge embedding systems into devices for real-time use. To address these limitations, we present a novel two-stream feature fusion "Tensor-Convolution and Convolution-Transformer Network" (TCCT-Net) architecture. To better learn the meaningful patterns in the temporal-spatial domain, we design a "CT" stream that integrates a hybrid convolutional-transformer. In parallel, to efficiently extract rich patterns from the temporal-frequency domain and boost processing speed, we introduce a "TC" stream that uses Continuous Wavelet Transform (CWT) to represent information in a 2D tensor form. Evaluated on the EngageNet dataset, the proposed method outperforms existing baselines, utilizing only two behavioral features (head pose rotations) compared to the 98 used in baseline models. Furthermore, comparative analysis shows TCCT-Net's architecture offers an order-of-magnitude improvement in inference speed compared to state-of-the-art image-based Recurrent Neural Network (RNN) methods. The code will be released at https://github.com/vedernikovphoto/TCCT_Net.
Paper Structure (17 sections, 5 equations, 3 figures, 4 tables)

This paper contains 17 sections, 5 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Method
  4. Temporal-spatial feature extraction stream
  5. Temporal-frequency feature extraction stream
  6. Feature fusion and classification module
  7. Experiments
  8. Dataset
  9. Preprocessing
  10. Data augmentation
  11. Training settings
  12. Results
  13. SOTA comparison
  14. EngageNet comparative analysis
  15. Speed performance
  16. ...and 2 more sections

Figures (3)

  • Figure 1: We introduce TCCT-Net, a novel architecture that outperforms the SOTA methods in accuracy and inference speed for the task of engagement analysis, showcasing superior efficiency and speed.
  • Figure 2: "Tensor-Convolution and Convolution-Transformer Network" (TCCT-Net) architecture diagram. TCCT-Net integrates temporal-spatial-frequency data obtained from behavioral feature signals. "TC" stream performs analysis on temporal-frequency behavioral features, while the "CT" stream focuses on their temporal-spatial analysis. The predictions from both streams are fused at the decision level.
  • Figure 3: Segmentation and Recombination (S&R) Augmentation. S&R tackles overfitting by segmenting behavioral feature signal data and recombining these segments, preserving essential features while introducing realistic variations.