Geometric Properties and Graph-Based Optimization of Neural Networks: Addressing Non-Linearity, Dimensionality, and Scalability

TL;DR

This paper investigates the geometric properties of neural networks and their graph-based representations to tackle non-linearity, dimensionality, and scalability. It combines three core avenues—advanced non-linear activations (polynomial neurons, RBFs, Leaky ReLU), dimensionality reduction with pruning, and scalable graph encoding via hierarchical gap methods—to optimize neural architectures. Empirical results across synthetic and real datasets show improved decision-boundary modeling, reduced computational burden, and substantial memory/time benefits for large graphs, supporting more robust and scalable models. The work advances interpretable, geometry-driven neural network design with practical impact for non-Euclidean data, large-scale graphs, and resource-constrained settings.

Abstract

Deep learning models are often considered black boxes due to their complex hierarchical transformations. Identifying suitable architectures is crucial for maximizing predictive performance with limited data. Understanding the geometric properties of neural networks involves analyzing their structure, activation functions, and the transformations they perform in high-dimensional space. These properties influence learning, representation, and decision-making. This research explores neural networks through geometric metrics and graph structures, building upon foundational work in arXiv:2007.06559. It addresses the limited understanding of geometric structures governing neural networks, particularly the data manifolds they operate on, which impact classification, optimization, and representation. We identify three key challenges: (1) overcoming linear separability limitations, (2) managing the dimensionality-complexity trade-off, and (3) improving scalability through graph representations. To address these, we propose leveraging non-linear activation functions, optimizing network complexity via pruning and transfer learning, and developing efficient graph-based models. Our findings contribute to a deeper understanding of neural network geometry, supporting the development of more robust, scalable, and interpretable models.

Figures (8)

• Figure 1: Chronological Overview Table of Key Advancements in Graph-Based Neural Networks
• Figure 2: Optimizing Geometric Properties of Neural Networks
• Figure 3: Pruning Transformation Example
• Figure 4: Decision Boundaries for Different Activations and Datasets
• Figure 5: Dimensionality Reduction and Pruning Techniques Compared Performance to Baseline