Beat Tracking as Object Detection

TL;DR

BeatFCOS reframes beat tracking as temporal object detection by modeling beats and downbeats as 1D intervals detected through a FCOS-like detector with a WaveBeat backbone. The approach replaces DBN post-processing with a data-driven NMS (and optionally Soft-NMS), introduces a left-edge–biased leftness loss, and uses a multi-scale FPN to capture temporal patterns. Key contributions include anchor-point design for 1D intervals, a leftness-based loss, and an end-to-end BeatFCOS pipeline that achieves competitive results across standard datasets, while simplifying hyperparameter tuning. The work demonstrates that object-detection techniques can effectively model rhythmic events with minimal adaptation, and suggests future directions in temporal adjacency constraints and EM-based learning to further improve beat alignment.

Abstract

Recent beat and downbeat tracking models (e.g., RNNs, TCNs, Transformers) output frame-level activations. We propose reframing this task as object detection, where beats and downbeats are modeled as temporal "objects." Adapting the FCOS detector from computer vision to 1D audio, we replace its original backbone with WaveBeat's temporal feature extractor and add a Feature Pyramid Network to capture multi-scale temporal patterns. The model predicts overlapping beat/downbeat intervals with confidence scores, followed by non-maximum suppression (NMS) to select final predictions. This NMS step serves a similar role to DBNs in traditional trackers, but is simpler and less heuristic. Evaluated on standard music datasets, our approach achieves competitive results, showing that object detection techniques can effectively model musical beats with minimal adaptation.

Figures (5)

• Figure 1: Full structure diagram of WaveBeat with BeatFCOS. Instead of the output of $C_8$ passing through two-channel Conv1D and Sigmoid layers to provide beat and downbeat activations, the $C_7$ and $C_8$ outputs are passed to $P_7$ and $P_8$, acting as the part that integrates the WaveBeat backbone with our BeatFCOS model.
• Figure 2: Beat and downbeat intervals are generated from a list of beats. Most notably, the downbeat is represented twice: once as a downbeat interval, and once as a regular beat interval.
• Figure 3: A beat interval of length 18 when downsampled to the base level, showing anchor points that overlap. Positive anchors are highlighted blue if the positive anchor point sub-box is at the left, or red if at the center. The coordinate values above the anchor points shows all regression targets.
• Figure 4: Distribution of IoU values between neighboring predicted intervals for beats (top, blue) and downbeats (bottom, red), grouped by confidence score ranges. High IoU values (on the right side of each histogram) represent redundant predictions, low values (on the left side of each histogram) indicate distinct predictions, and moderate values (around the middle of each histogram) reflect ambiguous cases. These distributions were calculated by aggregating the predictions from all the songs in the GTZAN test set tzanetakis2002musicalmarchand2015swing, which is known for its diverse set of genres: blues, classical, country, disco, hip-hop, jazz, metal, pop, reggae and rock.
• Figure :