Before AI Takes Over: Rethinking Nonlinear Signal Processing in Communications

TL;DR

This paper argues for integrating classical nonlinear signal processing with data-driven AI in communications by proposing a DCT-based nonlinear modeling framework built on a Hammerstein-type channel. It develops the DCT-based neuron, enabling compact, interpretable function approximation, and demonstrates its use in both flat-fading and frequency-selective channels via direct channel estimation and MDIR-based receivers. The key contributions include a rigorous analysis of polynomial versus DCT representations, an LMS-based learning scheme with favorable convergence properties, and a joint MDIR-DCT approach for nonlinear, memory-bearing channels with promising simulation results. The work advocates a principled, white-box approach that harmonizes learning and theory, paving the way for robust nonlinear compensation and transceiver design before AI completely dominates the field.

Abstract

There is an urgent reflection on traditional nonlinear signal processing methods in communications before Artificial Intelligence (AI) dominates the field. It implies a need to reassess or reinterpret established theories and tools, highlighting the tension between data-driven and model-based approaches. This paper calls for preserving valuable insights from classical signal processing while exploring how they can coexist or integrate with emerging AI methods.

Figures (9)

• Figure 1: Illustration of the relationship between the DFT and DCT for the sigmoid function representation.
• Figure 2: DCT-based neuron, where the LMS trains the coefficients.
• Figure 3: Performance of the DCT-based LMS algorithm for the square root function.
• Figure 4: Supervised estimation of the direct and the inverse nonlinear channel in AWGN.
• Figure 5: Comparison of inverse and direct channel estimation in an ideal noiseless channel ($\text{SNR}^{\text{pre}}=80$ dB) with $Q=6$ coefficients for both DCT models.