Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neuro-Symbolic Activation Discovery: Transferring Mathematical Structures from Physics to Ecology for Parameter-Efficient Neural Networks

Anas Hajbi

TL;DR

Neuro-Symbolic Activation Discovery introduces a two-phase framework where Genetic Programming discovers interpretable symbolic formulas from domain data and these formulas are injected as differentiable activations in lightweight neural nets. The study demonstrates a Geometric Transfer Phenomenon: activations learned on continuous physics data transfer to ecological classification, achieving comparable accuracy with 5–6x fewer parameters and a higher Parameter Efficiency Score, while transfer fails for continuous-to-discrete domains. A dedicated Parameter Efficiency Score, $E_{ ext{param}} = \frac{\text{AUC}}{\log_{10}(\text{Params})}$, quantifies efficiency gains across tasks. The work reveals boundary conditions for transferability and argues for domain-specific activation libraries to enable efficient scientific machine learning. Overall, the approach offers a blueprint for exploiting domain structure in activations to obtain parameter-efficient, interpretable models with practical transferability insights.

Abstract

Modern neural networks rely on generic activation functions (ReLU, GELU, SiLU) that ignore the mathematical structure inherent in scientific data. We propose Neuro-Symbolic Activation Discovery, a framework that uses Genetic Programming to extract interpretable mathematical formulas from data and inject them as custom activation functions. Our key contribution is the discovery of a Geometric Transfer phenomenon: activation functions learned from particle physics data successfully generalize to ecological classification, outperforming standard activations (ReLU, GELU, SiLU) in both accuracy and parameter efficiency. On the Forest Cover dataset, our Hybrid Transfer model achieves 82.4% accuracy with only 5,825 parameters, compared to 83.4% accuracy requiring 31,801 parameters for a conventional heavy network -- a 5.5x parameter reduction with only 1% accuracy loss. We introduce a Parameter Efficiency Score ($E_{param} = AUC / \log_{10}(Params)$) and demonstrate that lightweight hybrid architectures consistently achieve 18-21% higher efficiency than over-parameterized baselines. Crucially, we establish boundary conditions: while Physics to Ecology transfer succeeds (both involve continuous Euclidean measurements), Physics to Text transfer fails (discrete word frequencies require different mathematical structures). Our work opens pathways toward domain-specific activation libraries for efficient scientific machine learning.

Neuro-Symbolic Activation Discovery: Transferring Mathematical Structures from Physics to Ecology for Parameter-Efficient Neural Networks

TL;DR

Neuro-Symbolic Activation Discovery introduces a two-phase framework where Genetic Programming discovers interpretable symbolic formulas from domain data and these formulas are injected as differentiable activations in lightweight neural nets. The study demonstrates a Geometric Transfer Phenomenon: activations learned on continuous physics data transfer to ecological classification, achieving comparable accuracy with 5–6x fewer parameters and a higher Parameter Efficiency Score, while transfer fails for continuous-to-discrete domains. A dedicated Parameter Efficiency Score, , quantifies efficiency gains across tasks. The work reveals boundary conditions for transferability and argues for domain-specific activation libraries to enable efficient scientific machine learning. Overall, the approach offers a blueprint for exploiting domain structure in activations to obtain parameter-efficient, interpretable models with practical transferability insights.

Abstract

Modern neural networks rely on generic activation functions (ReLU, GELU, SiLU) that ignore the mathematical structure inherent in scientific data. We propose Neuro-Symbolic Activation Discovery, a framework that uses Genetic Programming to extract interpretable mathematical formulas from data and inject them as custom activation functions. Our key contribution is the discovery of a Geometric Transfer phenomenon: activation functions learned from particle physics data successfully generalize to ecological classification, outperforming standard activations (ReLU, GELU, SiLU) in both accuracy and parameter efficiency. On the Forest Cover dataset, our Hybrid Transfer model achieves 82.4% accuracy with only 5,825 parameters, compared to 83.4% accuracy requiring 31,801 parameters for a conventional heavy network -- a 5.5x parameter reduction with only 1% accuracy loss. We introduce a Parameter Efficiency Score () and demonstrate that lightweight hybrid architectures consistently achieve 18-21% higher efficiency than over-parameterized baselines. Crucially, we establish boundary conditions: while Physics to Ecology transfer succeeds (both involve continuous Euclidean measurements), Physics to Text transfer fails (discrete word frequencies require different mathematical structures). Our work opens pathways toward domain-specific activation libraries for efficient scientific machine learning.
Paper Structure (32 sections, 2 equations, 2 figures, 6 tables, 1 algorithm)

This paper contains 32 sections, 2 equations, 2 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Key Contributions
  3. Motivating Observation
  4. Related Work
  5. Activation Function Design
  6. Symbolic Regression and Equation Discovery
  7. Physics-Informed Neural Networks
  8. Methodology
  9. Phase 1: Symbolic Discovery via Genetic Programming
  10. GP Configuration
  11. Formula Generalization
  12. Phase 2: Hybrid Architecture
  13. Network Architectures
  14. Stability Considerations
  15. Evaluation Metrics
  16. ...and 17 more sections

Figures (2)

  • Figure 1: Discovered activation functions compared to ReLU and Swish baselines. (Left) HIGGS: oscillatory pattern capturing kinematic periodicity. (Center) Forest Cover: smooth wave-like structure. (Right) Spambase: linear growth with periodic modulation. Note that all discovered functions exhibit non-monotonic behavior, unlike ReLU.
  • Figure 2: Accuracy vs. model size across all datasets. Heavy models (red) achieve highest raw accuracy but lowest efficiency. Light baselines (blue) offer better trade-offs. Hybrid models show differentiated behavior: Transfer (green) excels for continuous domains (Forest Cover), while Specialist (orange) succeeds for domain-specific patterns (Spambase). Annotations highlight key success and failure cases.