A multi-purpose automatic editing system based on lecture semantics for remote education

TL;DR

This paper proposes an automatic multi-purpose editing system based on the lecture semantics, which can both direct the multiple video streams for real-time broadcasting and edit the optimal video offline for review purposes.

Abstract

Remote teaching has become popular recently due to its convenience and safety, especially under extreme circumstances like a pandemic. However, online students usually have a poor experience since the information acquired from the views provided by the broadcast platforms is limited. One potential solution is to show more camera views simultaneously, but it is technically challenging and distracting for the viewers. Therefore, an automatic multi-camera directing/editing system, which aims at selecting the most concerned view at each time instance to guide the attention of online students, is in urgent demand. However, existing systems mostly make simple assumptions and focus on tracking the position of the speaker instead of the real lecture semantics, and therefore have limited capacities to deliver optimal information flow. To this end, this paper proposes an automatic multi-purpose editing system based on the lecture semantics, which can both direct the multiple video streams for real-time broadcasting and edit the optimal video offline for review purposes. Our system directs the views by semantically analyzing the class events while following the professional directing rules, mimicking a human director to capture the regions of interest from the viewpoint of the onsite students. We conduct both qualitative and quantitative analyses to verify the effectiveness of the proposed system and its components.

Figures (11)

• Figure 1: The remote students can watch and manually switch between only two close-up views when taking classes online through online teaching software, e.g., Zoom.
• Figure 2: The illustration of our multi-view teaching environment. There are seven video streams, including close-up shots, medium shots, and long shots. Different shots can be used to convey different information..
• Figure 3: The overall architecture of the proposed editing system.
• Figure 4: The proposed skeleton-based event recognition architecture, consists of two GCN embedding branches and a cross-attention feature aggregation module.
• Figure 5: The skeleton topologies for two different situations. The first column is the predicted result by our method.