HorNets: Learning from Discrete and Continuous Signals with Routing Neural Networks

TL;DR

HorNets introduce a discrete–continuous routing neural network that learns from both discrete and continuous tabular data with scarce samples. By coupling a polyClip activation with a CatRouter-driven path selection and a discrete interaction module, the approach can explicitly model Horn clauses and propositional logic while maintaining strong predictive performance. Across synthetic benchmarks and 14 real biomedical datasets, HorNets achieve state-of-the-art or competitive results, offer interpretable rule-like interactions, and remain practical on standard hardware. The work advances neurosymbolic learning for high-dimensional, mixed-input data and provides open-source code and benchmarks for reproducibility.

Abstract

Construction of neural network architectures suitable for learning from both continuous and discrete tabular data is a challenging research endeavor. Contemporary high-dimensional tabular data sets are often characterized by a relatively small instance count, requiring data-efficient learning. We propose HorNets (Horn Networks), a neural network architecture with state-of-the-art performance on synthetic and real-life data sets from scarce-data tabular domains. HorNets are based on a clipped polynomial-like activation function, extended by a custom discrete-continuous routing mechanism that decides which part of the neural network to optimize based on the input's cardinality. By explicitly modeling parts of the feature combination space or combining whole space in a linear attention-like manner, HorNets dynamically decide which mode of operation is the most suitable for a given piece of data with no explicit supervision. This architecture is one of the few approaches that reliably retrieves logical clauses (including noisy XNOR) and achieves state-of-the-art classification performance on 14 real-life biomedical high-dimensional data sets. HorNets are made freely available under a permissive license alongside a synthetic generator of categorical benchmarks.

Figures (9)

• Figure 1: Overview of HorNets architecture. The CatRouter (Category Router) component decides whether to treat input as discrete or continuous. If discrete, CatInt layer is invoked, conducting factorization-based estimation of a subspace of interactions. If LinAtt is invoked, an efficient, element-wise PolyClip-based operation is considered. Both routes end up with a linear layer.
• Figure 2: Overview of the polyClip activation and its implications/interpretation when fully discretized (-1, 0, 1) range is considered. The curves were obtained by varying the $k$ parameter.
• Figure 3: Overview of algorithm performance (14 data sets). It can be observed that HorNets consistently rank among the top three performers, indicating the capability of this algorithm to operate on different problems consistently. Its main competitor in terms of tabular neural network learning, TabNet, failed to achieve high performance -- it is hypothesized this is due to high dimensionality/multiple classes of data considered.
• Figure 4: Friedman-Nemenyi test demonstrating the competitive performance of our method's F1-macro metric.
• Figure 5: Bayesian hierarchical t-test assessing differences between the HorNets-PolyClip variant and the TPOT AutoML model. The test indicates insignificant differences between HorNets-PolyClip and the state-of-the-art AutoML approach. The proposed approach is even marginally better.