Colony-Enhanced Recurrent Neural Architecture Search: Collaborative Ant-Based Optimization

TL;DR

This work introduces Collaborative Ant-based Neural Topology Search (CANTS-N), a multi-colony NAS framework that combines continuous ant-based topology search with PSO to evolve RNN topologies in a scalable, asynchronous HPC setting. By mapping discrete neural graphs to an unbounded 3D continuous space and enabling inter-colony knowledge sharing, CANTS-N achieves substantial performance gains over existing methods, demonstrated on a coal-fired plant dataset with 20 colonies and 1000 RNN structures each. The approach delivers an order-of-magnitude improvement in $MSE$ over baseline ACO/PSO methods, highlighting the potential of collaborative, colony-based optimization for NAS and NE. The paper also outlines future directions, including integrating Active Inference to endow agents with cognitive capabilities, potentially further boosting NAS efficiency and scalability.

Abstract

Crafting neural network architectures manually is a formidable challenge often leading to suboptimal and inefficient structures. The pursuit of the perfect neural configuration is a complex task, prompting the need for a metaheuristic approach such as Neural Architecture Search (NAS). Drawing inspiration from the ingenious mechanisms of nature, this paper introduces Collaborative Ant-based Neural Topology Search (CANTS-N), pushing the boundaries of NAS and Neural Evolution (NE). In this innovative approach, ant-inspired agents meticulously construct neural network structures, dynamically adapting within a dynamic environment, much like their natural counterparts. Guided by Particle Swarm Optimization (PSO), CANTS-N's colonies optimize architecture searches, achieving remarkable improvements in mean squared error (MSE) over established methods, including BP-free CANTS, BP CANTS, and ANTS. Scalable, adaptable, and forward-looking, CANTS-N has the potential to reshape the landscape of NAS and NE. This paper provides detailed insights into its methodology, results, and far-reaching implications.

Figures (4)

• Figure 1: Multi-Colony Flowchart
• Figure 2: Asynchronous Colonies
• Figure 3: CANTS-N Fitness: comparing CANTS-N (multi-colony) performance using the MSE of its colonies' best performing neural structures, to the best performing neural networks obtained from BP-CANTS, BP-free CANTS, and ANTS
• Figure 4: Colonies' Trajectories: colonies' characteristics explore the colony traits search space using PSO through exploration & exploitation