Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Training Deep Morphological Neural Networks as Universal Approximators

Konstantinos Fotopoulos, Petros Maragos

TL;DR

This paper addresses the challenge of training deep morphological neural networks (DMNNs) by showing that pure DMNNs are not universal approximators and introducing learnable linear activations between morphological layers. It proposes three constraint settings (Setting 1: MP-like, Setting 2: MP-SVD, Setting 3: Hybrid-MLP) to preserve sparsity while enabling universal approximation on compact domains, backed by universality results. The authors demonstrate through extensive experiments on MNIST, Fashion-MNIST, and CIFAR-10 that the proposed architectures are trainable, prunable, and can achieve competitive performance, with residual connections and dropout improving generalization. They further show morphological convolutions can benefit deeper networks, though gains vary by depth and task, highlighting the practical viability and potential of DMNNs for efficient representations and edge deployment.

Abstract

We investigate deep morphological neural networks (DMNNs). We demonstrate that despite their inherent non-linearity, "linear" activations are essential for DMNNs. To preserve their inherent sparsity, we propose architectures that constraint the parameters of the "linear" activations: For the first (resp. second) architecture, we work under the constraint that the majority of parameters (resp. learnable parameters) should be part of morphological operations. We improve the generalization ability of our networks via residual connections and weight dropout. Our proposed networks can be successfully trained, and are more prunable than linear networks. To the best of our knowledge, we are the first to successfully train DMNNs under such constraints. Finally, we propose a hybrid network architecture combining linear and morphological layers, showing empirically that the inclusion of morphological layers significantly accelerates the convergence of gradient descent with large batches.

Training Deep Morphological Neural Networks as Universal Approximators

TL;DR

This paper addresses the challenge of training deep morphological neural networks (DMNNs) by showing that pure DMNNs are not universal approximators and introducing learnable linear activations between morphological layers. It proposes three constraint settings (Setting 1: MP-like, Setting 2: MP-SVD, Setting 3: Hybrid-MLP) to preserve sparsity while enabling universal approximation on compact domains, backed by universality results. The authors demonstrate through extensive experiments on MNIST, Fashion-MNIST, and CIFAR-10 that the proposed architectures are trainable, prunable, and can achieve competitive performance, with residual connections and dropout improving generalization. They further show morphological convolutions can benefit deeper networks, though gains vary by depth and task, highlighting the practical viability and potential of DMNNs for efficient representations and edge deployment.

Abstract

We investigate deep morphological neural networks (DMNNs). We demonstrate that despite their inherent non-linearity, "linear" activations are essential for DMNNs. To preserve their inherent sparsity, we propose architectures that constraint the parameters of the "linear" activations: For the first (resp. second) architecture, we work under the constraint that the majority of parameters (resp. learnable parameters) should be part of morphological operations. We improve the generalization ability of our networks via residual connections and weight dropout. Our proposed networks can be successfully trained, and are more prunable than linear networks. To the best of our knowledge, we are the first to successfully train DMNNs under such constraints. Finally, we propose a hybrid network architecture combining linear and morphological layers, showing empirically that the inclusion of morphological layers significantly accelerates the convergence of gradient descent with large batches.
Paper Structure (32 sections, 16 theorems, 153 equations, 6 figures, 15 tables)

This paper contains 32 sections, 16 theorems, 153 equations, 6 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Prerequisites
  4. Tropical Algebra.
  5. Mathematical morphology.
  6. Deep morphological neural networks
  7. Max-plus MP-based DMNNs.
  8. ASFs and layers combining max-plus and min-plus MPs.
  9. DEP-based DMNNs.
  10. Our proposal.
  11. Setting 1.
  12. Improving generalization.
  13. Setting 2.
  14. Setting 3.
  15. Convolutional networks.
  16. ...and 17 more sections

Key Result

Theorem 3.1

For any network that only uses max-plus and min-plus MPs with input $\mathbf{x}\in \mathbb{R}^d$ and a single output $y(\mathbf{x})$, we have that $y(\mathbf{x})$ is Lipschitz continuous on $\mathbb{R}^d$ and a.e. it holds that either $\nabla y(\mathbf{x})=0$ or $\nabla y(\mathbf{x}) = \mathbf{e}_i

Figures (6)

  • Figure 1: Cases of differentiable and non-differentiable networks with respect to input and weights.
  • Figure 2: Different landscapes of morphological networks.
  • Figure 3: Distributions of MP and MPM networks.
  • Figure 4: Distribution of MNIST values.
  • Figure 5: Convergence rate of different models on MNIST and Fashion-MNIST for $6400$ batch size
  • ...and 1 more figures

Theorems & Definitions (36)

  • Theorem 3.1
  • Theorem 3.2
  • Theorem 3.3
  • Theorem 3.4
  • Theorem 3.5
  • Definition A.0: Universal approximator
  • Lemma A.1
  • proof
  • Lemma A.2
  • proof
  • ...and 26 more