Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Understanding the Link Between Modularity and Performance in Neural Networks for Reinforcement Learning

Humphrey Munn, Marcus Gallagher

TL;DR

It is found that the amount of network modularity for optimal performance is likely entangled in complex relationships between many other features of the network and problem environment, and direct optimisation or arbitrary designation of a suitable amount of modularity in neural networks may not be beneficial.

Abstract

Modularity has been widely studied as a mechanism to improve the capabilities of neural networks through various techniques such as hand-crafted modular architectures and automatic approaches. While these methods have sometimes shown improvements towards generalisation ability, robustness, and efficiency, the mechanisms that enable modularity to give performance advantages are unclear. In this paper, we investigate this issue and find that the amount of network modularity for optimal performance is likely entangled in complex relationships between many other features of the network and problem environment. Therefore, direct optimisation or arbitrary designation of a suitable amount of modularity in neural networks may not be beneficial. We used a classic neuroevolutionary algorithm which enables rich, automatic optimisation and exploration of neural network architectures and weights with varying levels of modularity. The structural modularity and performance of networks generated by the NeuroEvolution of Augmenting Topologies algorithm was assessed on three reinforcement learning tasks, with and without an additional modularity objective. The results of the quality-diversity optimisation algorithm, MAP-Elites, suggest intricate conditional relationships between modularity, performance, and other predefined network features.

Towards Understanding the Link Between Modularity and Performance in Neural Networks for Reinforcement Learning

TL;DR

It is found that the amount of network modularity for optimal performance is likely entangled in complex relationships between many other features of the network and problem environment, and direct optimisation or arbitrary designation of a suitable amount of modularity in neural networks may not be beneficial.

Abstract

Modularity has been widely studied as a mechanism to improve the capabilities of neural networks through various techniques such as hand-crafted modular architectures and automatic approaches. While these methods have sometimes shown improvements towards generalisation ability, robustness, and efficiency, the mechanisms that enable modularity to give performance advantages are unclear. In this paper, we investigate this issue and find that the amount of network modularity for optimal performance is likely entangled in complex relationships between many other features of the network and problem environment. Therefore, direct optimisation or arbitrary designation of a suitable amount of modularity in neural networks may not be beneficial. We used a classic neuroevolutionary algorithm which enables rich, automatic optimisation and exploration of neural network architectures and weights with varying levels of modularity. The structural modularity and performance of networks generated by the NeuroEvolution of Augmenting Topologies algorithm was assessed on three reinforcement learning tasks, with and without an additional modularity objective. The results of the quality-diversity optimisation algorithm, MAP-Elites, suggest intricate conditional relationships between modularity, performance, and other predefined network features.
Paper Structure (15 sections, 3 equations, 3 figures)

This paper contains 15 sections, 3 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Modularity
  4. Measuring Modularity
  5. NeuroEvolution of Augmenting Topologies (NEAT)
  6. Reinforcement Learning Environments
  7. MAP-Elites
  8. Methods
  9. MAP-Elites Feature Descriptors
  10. Results
  11. Characterising modularity and performance on standard NEAT optimisation
  12. Adding modularity reward to the fitness
  13. MAP-Elites performance landscape
  14. Discussion
  15. Conclusion

Figures (3)

  • Figure 1: OpenAI test environments. Left: BipedalWalker, Middle: Acrobot, Right: ContinuousLunarLander.
  • Figure 2: Average fitness (reward) and modularity (q-score) over generations of the standard NEAT learning algorithm. Each test is 20 runs, with 1 standard deviation confidence intervals.
  • Figure 3: MAP-Elites grid for BipedalWalker task showing the performance landscape over network modularity and other network features.