Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

StreamSense: Streaming Social Task Detection with Selective Vision-Language Model Routing

Han Wang, Deyi Ji, Lanyun Zhu, Jiebo Luo, Roy Ka-Wei Lee

TL;DR

This paper tackles real-time streaming social task detection (SSTD) by balancing accuracy and efficiency under partial, asynchronous multimodal evidence. It introduces StreamSense, a selective-routing framework that uses a lightweight Streaming Encoder for most timestamps and escalates difficult cases to a Vision-Language Model (VLM) expert, with an explicit deferral option for low-context moments. The encoder is trained with a cross-modal contrastive loss and an IoU-weighted cross-entropy to reduce boundary interference caused by segment-level supervision, while the VLM routing strategy controls when to invoke or defer, dramatically reducing latency and compute relative to VLM-only systems. Empirical results on MOSI, MOSEI, and HateClipSeg show StreamSense achieves higher accuracy than encoder-only baselines and substantially lowers VLM usage (approx. 27%) and latency (~0.3 s), demonstrating the value of selective escalation and deferral for practical streaming social monitoring. The work advances real-time multimodal moderation and analytics by providing a model-agnostic, efficiency-conscious framework with robust performance under streaming constraints.

Abstract

Live streaming platforms require real-time monitoring and reaction to social signals, utilizing partial and asynchronous evidence from video, text, and audio. We propose StreamSense, a streaming detector that couples a lightweight streaming encoder with selective routing to a Vision-Language Model (VLM) expert. StreamSense handles most timestamps with the lightweight streaming encoder, escalates hard/ambiguous cases to the VLM, and defers decisions when context is insufficient. The encoder is trained using (i) a cross-modal contrastive term to align visual/audio cues with textual signals, and (ii) an IoU-weighted loss that down-weights poorly overlapping target segments, mitigating label interference across segment boundaries. We evaluate StreamSense on multiple social streaming detection tasks (e.g., sentiment classification and hate content moderation), and the results show that StreamSense achieves higher accuracy than VLM-only streaming while only occasionally invoking the VLM, thereby reducing average latency and compute. Our results indicate that selective escalation and deferral are effective primitives for understanding streaming social tasks. Code is publicly available on GitHub.

StreamSense: Streaming Social Task Detection with Selective Vision-Language Model Routing

TL;DR

This paper tackles real-time streaming social task detection (SSTD) by balancing accuracy and efficiency under partial, asynchronous multimodal evidence. It introduces StreamSense, a selective-routing framework that uses a lightweight Streaming Encoder for most timestamps and escalates difficult cases to a Vision-Language Model (VLM) expert, with an explicit deferral option for low-context moments. The encoder is trained with a cross-modal contrastive loss and an IoU-weighted cross-entropy to reduce boundary interference caused by segment-level supervision, while the VLM routing strategy controls when to invoke or defer, dramatically reducing latency and compute relative to VLM-only systems. Empirical results on MOSI, MOSEI, and HateClipSeg show StreamSense achieves higher accuracy than encoder-only baselines and substantially lowers VLM usage (approx. 27%) and latency (~0.3 s), demonstrating the value of selective escalation and deferral for practical streaming social monitoring. The work advances real-time multimodal moderation and analytics by providing a model-agnostic, efficiency-conscious framework with robust performance under streaming constraints.

Abstract

Live streaming platforms require real-time monitoring and reaction to social signals, utilizing partial and asynchronous evidence from video, text, and audio. We propose StreamSense, a streaming detector that couples a lightweight streaming encoder with selective routing to a Vision-Language Model (VLM) expert. StreamSense handles most timestamps with the lightweight streaming encoder, escalates hard/ambiguous cases to the VLM, and defers decisions when context is insufficient. The encoder is trained using (i) a cross-modal contrastive term to align visual/audio cues with textual signals, and (ii) an IoU-weighted loss that down-weights poorly overlapping target segments, mitigating label interference across segment boundaries. We evaluate StreamSense on multiple social streaming detection tasks (e.g., sentiment classification and hate content moderation), and the results show that StreamSense achieves higher accuracy than VLM-only streaming while only occasionally invoking the VLM, thereby reducing average latency and compute. Our results indicate that selective escalation and deferral are effective primitives for understanding streaming social tasks. Code is publicly available on GitHub.
Paper Structure (28 sections, 9 equations, 4 figures, 5 tables)

This paper contains 28 sections, 9 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Social Task Detection
  4. Streaming Video Understanding
  5. Streaming Social Task Detection
  6. Task Formulation
  7. Challenge I: Balancing Performance and Efficiency
  8. Challenge II: Absence of Stream-Level Annotation
  9. Methodology
  10. Streaming Encoder (Classifier)
  11. Selective Routing to the VLM Expert
  12. Deferral Policy
  13. Experiments
  14. Experimental Settings
  15. Datasets.
  16. ...and 13 more sections

Figures (4)

  • Figure 1: Comparison of (1) offline versus streaming video understanding and (2) general versus social task detection. Neg = Negative, Hate = Hate Speech.
  • Figure 2: Comparison of segment-level and stream-level annotations using sentiment classification as an example. Pos = Positive, Neg = Negative, Unk = Unknown
  • Figure 3: Overview of the proposed StreamSense framework.
  • Figure 4: StreamSense model performance for MOSI dataset under the threshold-based VLM-Invocation and Prediction Defer strategy, evaluated across varying MaxEnc (0–30) and MaxDefer (0–10). MaxEnc: Maximum Encoder Allowance Steps, MaxDefer: Maximum Deferral Allowance Steps . Metrics: VLM-Invoc = VLM Invocation Rate, VLM-Defer = VLM Defer Rate, M-F1 = Macro-F1.