Hybrid Real- and Complex-valued Neural Network Architecture

TL;DR

This work addresses the inefficiencies of real-valued neural networks when handling inherently complex-valued data by proposing a Hybrid Real- and Complex-valued Neural Network (HNN) that jointly processes real and complex information through domain conversions and dedicated complex activations. A dedicated neural architecture search framework guides the design of inter-domain pathways and blocks, achieving parameter-efficient models without sacrificing performance. Empirical results on AudioMNIST show that HNNs reduce cross-entropy loss and parameters relative to RVNNs, with robustness to noise and partial data, illustrating the practical potential for hybrid processing in signal domains. The approach sets the stage for broader adoption of partially complex-valued processing in neural networks and opens avenues for extensions to other architectures and complex-valued tasks.

Abstract

We propose a \emph{hybrid} real- and complex-valued \emph{neural network} (HNN) architecture, designed to combine the computational efficiency of real-valued processing with the ability to effectively handle complex-valued data. We illustrate the limitations of using real-valued neural networks (RVNNs) for inherently complex-valued problems by showing how it learnt to perform complex-valued convolution, but with notable inefficiencies stemming from its real-valued constraints. To create the HNN, we propose to use building blocks containing both real- and complex-valued paths, where information between domains is exchanged through domain conversion functions. We also introduce novel complex-valued activation functions, with higher generalisation and parameterisation efficiency. HNN-specific architecture search techniques are described to navigate the larger solution space. Experiments with the AudioMNIST dataset demonstrate that the HNN reduces cross-entropy loss and consumes less parameters compared to an RVNN for all considered cases. Such results highlight the potential for the use of partially complex-valued processing in neural networks and applications for HNNs in many signal processing domains.

Figures (10)

• Figure 1: Non-ordered weights for the first convolutional layer.
• Figure 2: Plot of reordered and normalized weights for each layer's convolutions for the RVNN experiment. The arrows indicate how the output of each layer is followed by the input of another. Distinct regions in the weight patterns are highlighted with dashed blue lines, presenting a letter as its label. Refer to the colour bar in Figure \ref{['fig:nonordered_weights']}.
• Figure 3: Detailed schematic of the trained and decoded RVNN used for the experiment, given weight regions defined in Figure \ref{['fig:layers_regions']}. The complex path shows the activations that represent the path where complex-valued data is conveyed in various forms, learned by the RVNN. When a number is indicated after the input, a layer, or a summation, it indicates size.
• Figure 4: Multi-phase real-valued functions observed in the 4-, 3- and 2-phase indication in Figure \ref{['fig:experiment_schematic']}.
• Figure 5: Detail view of reordered weights of the first convolutional layers in the RVNN network considered for the experiment.