TF-SepNet: An Efficient 1D Kernel Design in CNNs for Low-Complexity Acoustic Scene Classification

TL;DR

TF-SepNet addresses the need for low-complexity acoustic scene classification by introducing Time-Frequency Separate Convolutions that apply independent $3\times1$ and $1\times3$ kernels to the input spectrogram and fuse the resulting features for classification. The architecture follows a BC-ResNet-like backbone with adaptive residual normalization, enabling a larger effective receptive field while reducing parameters and multiply-accumulate operations. On TAU Urban Acoustic Scene 2022 Mobile data, TF-SepNet achieves higher accuracy than state-of-the-art consecutive-kernel models with substantially lower computation, and ablation/ERF analyses corroborate the benefits of the separate time and frequency paths. These results demonstrate that time-frequency separated convolutions offer a principled, efficient approach for ASC and may extend to other audio recognition tasks.

Abstract

Recent studies focus on developing efficient systems for acoustic scene classification (ASC) using convolutional neural networks (CNNs), which typically consist of consecutive kernels. This paper highlights the benefits of using separate kernels as a more powerful and efficient design approach in ASC tasks. Inspired by the time-frequency nature of audio signals, we propose TF-SepNet, a CNN architecture that separates the feature processing along the time and frequency dimensions. Features resulted from the separate paths are then merged by channels and directly forwarded to the classifier. Instead of the conventional two dimensional (2D) kernel, TF-SepNet incorporates one dimensional (1D) kernels to reduce the computational costs. Experiments have been conducted using the TAU Urban Acoustic Scene 2022 Mobile development dataset. The results show that TF-SepNet outperforms similar state-of-the-arts that use consecutive kernels. A further investigation reveals that the separate kernels lead to a larger effective receptive field (ERF), which enables TF-SepNet to capture more time-frequency features.

Figures (3)

• Figure 1: Simplified diagrams of two 1D-kernel-based design approaches in CNNs. $x$ denotes the input features.
• Figure 2: Left: Visualization of the Time-Frequency Separate Convolutions (TF-SepConvs) module. Right: Transformation of features maps. DWConv represents the depthwise convolution and PWConv denotes the pointwise convolution. Freq/Temp prefix denotes the 1D operation on the frequential or temporal axis. Shuffle indicates the channel shuffle unit Zhang2018. The dashed line or box means corresponding operation only exists when the channel number changes. The input feature $x$ is in $\mathbb{R}^{C \times F \times T}$, where $C, F, T$ respectively denotes channel, frequency and time dimensions.
• Figure 3: Visualization of Effective Receptive Fields (ERF). The color intensity at each point signifies the contribution score of the corresponding pixel in the input image to the central point of the feature map generated by the final layer. A broader and darker region indicates a larger ERF.