Generational Replacement and Learning for High-Performing and Diverse Populations in Evolvable Robots

TL;DR

The paper tackles the challenge of co-optimizing robot morphology and control, which can cause control–morphology mismatch and reduced diversity. It proposes combining intra-life learning of controllers with generational replacement, and compares elitist versus generational survivors under both learning conditions. The results show that generational replacement with learning maintains high diversity and achieves strong performance, but the observed benefit of learning is highly sensitive to whether performance is measured by function evaluations or generations. The findings inform evaluation practices and suggest that a post-evolution learning phase can close the gap to fully learned controllers in evolvable robotic systems.

Abstract

Evolutionary Robotics offers the possibility to design robots to solve a specific task automatically by optimizing their morphology and control together. However, this co-optimization of body and control is challenging, because controllers need some time to adapt to the evolving morphology - which may make it difficult for new and promising designs to enter the evolving population. A solution to this is to add intra-life learning, defined as an additional controller optimization loop, to each individual in the evolving population. A related problem is the lack of diversity often seen in evolving populations as evolution narrows the search down to a few promising designs too quickly. This problem can be mitigated by implementing full generational replacement, where offspring robots replace the whole population. This solution for increasing diversity usually comes at the cost of lower performance compared to using elitism. In this work, we show that combining such generational replacement with intra-life learning can increase diversity while retaining performance. We also highlight the importance of performance metrics when studying learning in morphologically evolving robots, showing that evaluating according to function evaluations versus according to generations of evolution can give different conclusions.

Figures (7)

• Figure 1: Overview of an evolutionary loop for morphology optimization (the blue lines) with a learning loop for controller optimization (the red lines). When comparing over generations, we compare over the blue loop, while comparing over function evaluations also takes the red loop into account.
• Figure 2: Overview of the robot's control and morphology. There are three types of modules, and every robot always has one head module. The head module has four attachment points, the block module has six (it includes up and down) and the joint module has two. Symmetry is ensured by making the left and right parts of the core module identical. Each joint module is controlled by its own sine wave parameters, with symmetrical parts sharing the same set of parameters.
• Figure 3: To find a suitable learning budget for Bayesian optimization, we tested how fast robot performance increases during the search. We tested Bayesian optimization on 100 instances of two types of robots and plotted their performance relative to their maximum performance. The two types of robots are random robots and the best robots of a preliminary evolutionary search. We plot two lines for the best robots, one with a random first sample and one with the first sample of the evolutionary search. After 30 samples, on average already 50% of the robots' potential is reached, this learning budget is therefore chosen for the evolution-with-learning setting.
• Figure 4: The rough environment with an example robot. The environment is generated by adding noise to a flat height map, and the height map is the same for every evaluation. The goal is to move forward as far as possible in 30 simulated seconds.
• Figure 5: The performance (objective value) and diversity (tree edit distance) compared over function evaluations. The performance shows the best objective value so far. The values at the end are indicated for clarity. The lines are averaged over runs, and the shaded areas are the standard deviation. The initial jump in performance and drop in diversity can be explained by the algorithm initially filtering out bad morphologies.