Real-Time Semantic Background Subtraction

TL;DR

RT-SBS tackles real-time background subtraction by combining a real-time background model with per-pixel semantic information that may arrive at slower rates. It extends SBS with a per-pixel change detector $C_t(x,y)$ to decide when prior semantic data $S_{t^*}(x,y)$ remains valid and introduces a semantic feedback loop that substitutes the BGS update with the RT-SBS decision $D_t(x,y)$. The approach yields state-of-the-art performance among real-time unsupervised BGS methods on CDNet 2014, approaching non-real-time SBS, and is implemented with both CPU and GPU code for practical use. These contributions enable accurate, real-time foreground segmentation in dynamic scenes while leveraging semantic cues without requiring frame-by-frame semantic computation.

Abstract

Semantic background subtraction SBS has been shown to improve the performance of most background subtraction algorithms by combining them with semantic information, derived from a semantic segmentation network. However, SBS requires high-quality semantic segmentation masks for all frames, which are slow to compute. In addition, most state-of-the-art background subtraction algorithms are not real-time, which makes them unsuitable for real-world applications. In this paper, we present a novel background subtraction algorithm called Real-Time Semantic Background Subtraction (denoted RT-SBS) which extends SBS for real-time constrained applications while keeping similar performances. RT-SBS effectively combines a real-time background subtraction algorithm with high-quality semantic information which can be provided at a slower pace, independently for each pixel. We show that RT-SBS coupled with ViBe sets a new state of the art for real-time background subtraction algorithms and even competes with the non real-time state-of-the-art ones. Note that we provide python CPU and GPU implementations of RT-SBS at https://github.com/cioppaanthony/rt-sbs.

Figures (3)

• Figure 1: Our novel background subtraction algorithm, called ${\color{red}\text{RT-SBS}},$ combines a $\text{BGS}$ algorithm with semantic information in real time. Semantic information is slow to compute and is only available for some frames, but $\text{RT-SBS}$ reuses previous semantic information when appropriate.
• Figure 2: Overall $F_{1}$ scores obtained with $\text{RT-SBS}$ (built upon ViBe) as a function of the semantic frame rate $\text{$X\mathalpha{:}1$}$ of $\text{RT-SBS}$ with feedback, $\text{RT-SBS}$ without feedback, the first heuristic, the second heuristic, and the original ViBe algorithm.
• Figure 3: Qualitative evaluation of $\text{RT-SBS}$. From left to right: the original color image, the ground truth, the background subtraction of ViBe, $\text{RT-SBS}$, and $\text{RT-SBS}$ with a feedback.