Accelerating Automatic Differentiation of Direct Form Digital Filters

TL;DR

The paper addresses the challenge of differentiating direct-form and transposed-direct-form digital filters within neural networks, where recursive filters incur AD overhead. It derives a general, closed-form backward pass in the time-domain state-space representation, including gradients for initial conditions, and shows the backward pass is itself a state-space recursion. The authors implement C++/CUDA PyTorch extensions that register the filter and its gradient as native operators, enabling parallel time processing and achieving substantial speedups over naive Python and over frequency-domain baselines for low-order filters. This work enables exact time-domain differentiation in differentiable signal-processing pipelines, with practical implications for efficient end-to-end models and avenues for parameter-varying extensions and forward-mode differentiation.

Abstract

We introduce a general formulation for automatic differentiation through direct form filters, yielding a closed-form backpropagation that includes initial condition gradients. The result is a single expression that can represent both the filter and its gradients computation while supporting parallelism. C++/CUDA implementations in PyTorch achieve at least 1000x speedup over naive Python implementations and consistently run fastest on the GPU. For the low-order filters commonly used in practice, exact time-domain filtering with analytical gradients outperforms the frequency-domain method in terms of speed. The source code is available at https://github.com/yoyolicoris/philtorch.

Figures (1)

• Figure 1: Runtime of the recursion Eq. \ref{['eq:ss_recursion']} and its backpropagation with various signal lengths $N$ and implementations.