BILLNET: A Binarized Conv3D-LSTM Network with Logic-gated residual architecture for hardware-efficient video inference

TL;DR

This work targets the hardware bottlenecks of video inference by introducing BILLNET, a compact binarized Conv3D-LSTM that employs Conv3D factorization and a 3D MUX-OR residual to maintain binary-compatibleSkip connections. A five-stage multi-stage quantization training pipeline enables fully quantized weights and activations, including LSTM components, while BitShift Normalization replaces BatchNorm for hardware friendliness. On the Jester dataset, BILLNET achieves competitive accuracy with significantly reduced memory and computation (GBOPs) compared to other resource-efficient models, and provides a hardware-ready path for FPGA/ASIC deployment. The combination of CF, MOR, and fully quantized LSTM demonstrates practical trade-offs between accuracy and hardware efficiency for embedded video inference.

Abstract

Long Short-Term Memory (LSTM) and 3D convolution (Conv3D) show impressive results for many video-based applications but require large memory and intensive computing. Motivated by recent works on hardware-algorithmic co-design towards efficient inference, we propose a compact binarized Conv3D-LSTM model architecture called BILLNET, compatible with a highly resource-constrained hardware. Firstly, BILLNET proposes to factorize the costly standard Conv3D by two pointwise convolutions with a grouped convolution in-between. Secondly, BILLNET enables binarized weights and activations via a MUX-OR-gated residual architecture. Finally, to efficiently train BILLNET, we propose a multi-stage training strategy enabling to fully quantize LSTM layers. Results on Jester dataset show that our method can obtain high accuracy with extremely low memory and computational budgets compared to existing Conv3D resource-efficient models.

Figures (6)

• Figure 1: Top-level architecture description of BILLNET with Convolutional Factorization (CF) and MUX-OR Residual (MOR) Block. Here $n$ is the parameters controlling the number of output feature maps, g-GConv is Grouped Convolution with g groups, MP and GAP stand for Max Pooling and Global Average Pooling.
• Figure 2: The operation of the channel-wise MUX gate with feature maps extracted during inference of a test sample. The TGAP is implemented by a bitcount followed by an integer-to-integer comparison, where the threshold is equal to one half of the spatial resolution ($\frac{6 \times 8}{2}=24$).
• Figure 3: Computational graph of the proposed Quantized LSTM.
• Figure 4: Training curves (CCE loss and accuracy) of BILLNET $2\times$ throughout all 5 training stages.
• Figure 5: Weight memory (Mb) and computational costs (GBOPs $\sim$$10^9$ BOPs BOP) versus Top-1 Accuracy. Edge-less stars are for BILLNET with $g$=$2$ and $m$=$n$, edged stars are for $m$=$n/2$ and $g=\frac{n}{16}$ (with $n\in \{64, 128\}$).