ML-Enabled Deformable Matched Filters for Bandlimitation Compensation in Free-Space Optics

TL;DR

This work tackles CAP modulation in bandwidth-limited free-space optical links by introducing a hybrid deformable matched filter that learns a residual complex deformation $\Delta \mathbf{h} \in \mathbb{C}^L$ from a compact feature set $\mathbf{f} \in \mathbb{R}^{16}$ to adapt fixed CAP filters of length $L$ (with $L=192$). Trained with an end-to-end differentiable loss that combines the error vector magnitude $\mathcal{L}_{\rm EVM}$ and smoothness regularisers, the method delivers substantial EVM reductions under severe bandwidth constraint, validated in hardware-in-the-loop experiments. The approach preserves classical receiver structure while providing adaptive pulse-shape compensation, and it gracefully defaults to conventional matched filtering when channel conditions are favorable, offering a practical path for robust CAP communications in bandwidth-limited scenarios. Potential extensions include joint transmit/receive filter optimisation to further enhance link efficiency without adding receiver latency.

Abstract

This paper proposes a neural-network-assisted deformable matched filtering framework for carrier-less amplitude and phase (CAP) modulation operating under bandwidth-limited channel conditions. Instead of replacing the analytically derived CAP matched filter, the proposed receiver learns a residual deformation of the nominal matched filter based on a compact set of physically motivated signal features extracted from the received waveform. A total of 16 time-domain, frequency-domain, and memory-related features are used to provide a low-dimensional representation of bandwidth-induced pulse distortion. These features are mapped by a fully connected neural network to complex-valued matched filter coefficients, enabling adaptive pulse-shape compensation prior to symbol-rate sampling. The network is trained end-to-end using a differentiable loss function based on error vector magnitude (EVM). Experimental results obtained using a hardware-in-the-loop CAP transmission system demonstrate that the proposed deformable matched filter significantly outperforms conventional fixed matched filtering under severe bandwidth constraints, without requiring decision feedback or increasing receiver latency.

Figures (6)

• Figure 1: Block diagram showing the high level architecture of the system.
• Figure 2: Photograph of the test setup.
• Figure 3: An illustrative example of how the neural network connects the features to the filter coefficients.
• Figure 4: EVM vs received optical power for 4-QAM across eight $\omega_n$ values. Solid: conventional MF; dashed: proposed NN. Region A shows maximum improvement.
• Figure 5: Imaginary component of the deformed matched filters learned by the neural network for different normalised cut-off frequencies at OD = 0.1. The ideal matched filter (dashed black line) is shown for reference.