Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Task-Driven Exploration: Decoupling and Inter-Task Feedback for Joint Moment Retrieval and Highlight Detection

Jin Yang, Ping Wei, Huan Li, Ziyang Ren

TL;DR

This work addresses the joint problem of moment retrieval (MR) and highlight detection (HD) by introducing TaskWeave, a task-driven top-down framework that decouples task-specific and common representations and adds an inter-task feedback loop to exchange guidance via masks. The architecture follows a DETR-like pipeline where a task-decoupled unit produces MR and HD features, and two decoders enable cross-task refinement, with a principled, learnable joint loss that balances MR and HD via uncertainties $\delta_{mr}$ and $\delta_{hd}$. Key innovations include the task-decoupled unit (shared plus task-specific experts), moment-to-mask and highlightness-to-mask feedback, and a dynamic, task-dependent loss, all validated on QVHighlights, Charades-STA, and TVSum to achieve state-of-the-art or competitive results. These contributions demonstrate that explicit modeling of task specificity and inter-task influence, coupled with adaptive optimization, can significantly improve performance on complex, multi-task video understanding tasks, with practical impact for robust video search and content analysis tooling.

Abstract

Video moment retrieval and highlight detection are two highly valuable tasks in video understanding, but until recently they have been jointly studied. Although existing studies have made impressive advancement recently, they predominantly follow the data-driven bottom-up paradigm. Such paradigm overlooks task-specific and inter-task effects, resulting in poor model performance. In this paper, we propose a novel task-driven top-down framework TaskWeave for joint moment retrieval and highlight detection. The framework introduces a task-decoupled unit to capture task-specific and common representations. To investigate the interplay between the two tasks, we propose an inter-task feedback mechanism, which transforms the results of one task as guiding masks to assist the other task. Different from existing methods, we present a task-dependent joint loss function to optimize the model. Comprehensive experiments and in-depth ablation studies on QVHighlights, TVSum, and Charades-STA datasets corroborate the effectiveness and flexibility of the proposed framework. Codes are available at https://github.com/EdenGabriel/TaskWeave.

Task-Driven Exploration: Decoupling and Inter-Task Feedback for Joint Moment Retrieval and Highlight Detection

TL;DR

This work addresses the joint problem of moment retrieval (MR) and highlight detection (HD) by introducing TaskWeave, a task-driven top-down framework that decouples task-specific and common representations and adds an inter-task feedback loop to exchange guidance via masks. The architecture follows a DETR-like pipeline where a task-decoupled unit produces MR and HD features, and two decoders enable cross-task refinement, with a principled, learnable joint loss that balances MR and HD via uncertainties and . Key innovations include the task-decoupled unit (shared plus task-specific experts), moment-to-mask and highlightness-to-mask feedback, and a dynamic, task-dependent loss, all validated on QVHighlights, Charades-STA, and TVSum to achieve state-of-the-art or competitive results. These contributions demonstrate that explicit modeling of task specificity and inter-task influence, coupled with adaptive optimization, can significantly improve performance on complex, multi-task video understanding tasks, with practical impact for robust video search and content analysis tooling.

Abstract

Video moment retrieval and highlight detection are two highly valuable tasks in video understanding, but until recently they have been jointly studied. Although existing studies have made impressive advancement recently, they predominantly follow the data-driven bottom-up paradigm. Such paradigm overlooks task-specific and inter-task effects, resulting in poor model performance. In this paper, we propose a novel task-driven top-down framework TaskWeave for joint moment retrieval and highlight detection. The framework introduces a task-decoupled unit to capture task-specific and common representations. To investigate the interplay between the two tasks, we propose an inter-task feedback mechanism, which transforms the results of one task as guiding masks to assist the other task. Different from existing methods, we present a task-dependent joint loss function to optimize the model. Comprehensive experiments and in-depth ablation studies on QVHighlights, TVSum, and Charades-STA datasets corroborate the effectiveness and flexibility of the proposed framework. Codes are available at https://github.com/EdenGabriel/TaskWeave.
Paper Structure (13 sections, 9 equations, 4 figures, 7 tables)

This paper contains 13 sections, 9 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Overview
  5. Task-decoupled Unit
  6. Inter-task Feedback Mechanism
  7. Task-dependent Joint Loss
  8. Experiments
  9. Experimental Setup
  10. Implementation Details
  11. Comparison with State-of-the-arts
  12. Ablation Studies and Discussions
  13. Conclusion

Figures (4)

  • Figure 1: Although the positive temporal intervals of moment retrieval and highlight detection exhibit high overlap, they pursue different objectives.
  • Figure 2: The overall pipeline of the proposed task-driven model TaskWeave. We propose the task-decoupled unit to capture task-specific and common features. Various experts can adopt different network implementations, showcasing the flexibility of the model. Inter-task feedback mechanism is designed to investigate the influence between both tasks. There are two feedback manners: Moment-guided and Highlightness-guided feedback. The principled task-dependent joint loss is introduced for jointly optimize the model.
  • Figure 3: Illustration of the inter-task feedback mechanism. (a) moment-guided feedback manner. (b) highlightness-guided feedback manner.
  • Figure 4: Qualitative results on the QVHighlights for Ground-Truth, Moment-DETR, QD-DETR and our method. The predicted moments and saliency scores are illustrated through intervals and lines.