Descriptive complexity for neural networks via Boolean networks
Veeti Ahvonen, Damian Heiman, Antti Kuusisto
TL;DR
This work establishes a strong descriptive-complexity bridge between neural networks and symbolic logics by translating between general recurrent FF-enabled NNs, Boolean network logic (BNL), SC/MSC variants, and self-feeding circuits. It proves that both directions of translation incur only polynomial or linear-size blow-ups and polylogarithmic delays in key regimes, and it shows that regular activation functions allow broad substitution across architectures, including linear activations via floating-point arithmetic. The framework yields FFNN and FFBNL translations with comparable complexity, and demonstrates a deep equivalence between symbolic and non-symbolic computation models for finite input spaces. The results open avenues for circuit- and logic-based reasoning about neural networks and suggest future work on randomized models and fixed-point arithmetic extensions.
Abstract
We investigate the expressive power of neural networks from the point of view of descriptive complexity. We study neural networks that use floating-point numbers and piecewise polynomial activation functions from two perspectives: 1) the general scenario where neural networks run for an unlimited number of rounds and have unrestricted topologies, and 2) classical feedforward neural networks that have the topology of layered acyclic graphs and run for only a constant number of rounds. We characterize these neural networks via Boolean networks formalized via a recursive rule-based logic. In particular, we show that the sizes of the neural networks and the corresponding Boolean rule formulae are polynomially related. In fact, in the direction from Boolean rules to neural networks, the blow-up is only linear. Our translations result in a time delay, which is the number of rounds that it takes to simulate a single computation step. In the translation from neural networks to Boolean rules, the time delay of the resulting formula is polylogarithmic in the size of the neural network. In the converse translation, the time delay of the neural network is linear in the formula size. Ultimately, we obtain translations between neural networks, Boolean networks, the diamond-free fragment of modal substitution calculus, and a class of recursive Boolean circuits. Our translations offer a method, for almost any activation function F, of translating any neural network in our setting into an equivalent neural network that uses F at each node. This even includes linear activation functions, which is possible due to using floats rather than actual reals!