Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalized Early Stopping in Evolutionary Direct Policy Search

Etor Arza, Leni K. Le Goff, Emma Hart

TL;DR

This work proposes an early stopping method for direct policy search that only looks at the objective value at each timestep and requires no problem-specific knowledge, and is tested in five direct policy search environments drawn from games, robotics, and classic control domains.

Abstract

Lengthy evaluation times are common in many optimization problems such as direct policy search tasks, especially when they involve conducting evaluations in the physical world, e.g. in robotics applications. Often when evaluating solution over a fixed time period it becomes clear that the objective value will not increase with additional computation time (for example when a two wheeled robot continuously spins on the spot). In such cases, it makes sense to stop the evaluation early to save computation time. However, most approaches to stop the evaluation are problem specific and need to be specifically designed for the task at hand. Therefore, we propose an early stopping method for direct policy search. The proposed method only looks at the objective value at each time step and requires no problem specific knowledge. We test the introduced stopping criterion in five direct policy search environments drawn from games, robotics and classic control domains, and show that it can save up to 75% of the computation time. We also compare it with problem specific stopping criteria and show that it performs comparably, while being more generally applicable.

Generalized Early Stopping in Evolutionary Direct Policy Search

TL;DR

This work proposes an early stopping method for direct policy search that only looks at the objective value at each timestep and requires no problem-specific knowledge, and is tested in five direct policy search environments drawn from games, robotics, and classic control domains.

Abstract

Lengthy evaluation times are common in many optimization problems such as direct policy search tasks, especially when they involve conducting evaluations in the physical world, e.g. in robotics applications. Often when evaluating solution over a fixed time period it becomes clear that the objective value will not increase with additional computation time (for example when a two wheeled robot continuously spins on the spot). In such cases, it makes sense to stop the evaluation early to save computation time. However, most approaches to stop the evaluation are problem specific and need to be specifically designed for the task at hand. Therefore, we propose an early stopping method for direct policy search. The proposed method only looks at the objective value at each time step and requires no problem specific knowledge. We test the introduced stopping criterion in five direct policy search environments drawn from games, robotics and classic control domains, and show that it can save up to 75% of the computation time. We also compare it with problem specific stopping criteria and show that it performs comparably, while being more generally applicable.
Paper Structure (19 sections, 5 equations, 14 figures, 2 tables, 1 algorithm)

This paper contains 19 sections, 5 equations, 14 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Generalized Early Stopping for Direct Policy Search (GESP)
  4. Generalized Early Stopping for Direct Policy S.earch
  5. Applicability on direct policy search
  6. Experimentation
  7. Classic control tasks (classic control)
  8. Playing Super Mario (super mario)
  9. Tasks in the Mujoco environment (mujoco)
  10. NIPES within the ARE framework (NIPES explore)
  11. Robotics, Evolution and Modularity (L-System)
  12. On the parameter $t_{grace}$
  13. Performance with respect to $t_{grace}$
  14. A closer look on $t_{grace}$
  15. Discussion and future work
  16. ...and 4 more sections

Figures (14)

  • Figure 1: The objective value of the agents with and without GESP with respect to computation time (classic control).
  • Figure 2: Ratio of solutions evaluated with and without GESP in the same optimization time. A higher value indicates that GESP was able to evaluate more solutions in the same time.
  • Figure 3: The objective value of the agents with respect to computation time in super mario with GESP, with the problem specific stopping criterion and without additional stopping criterion. Levels from top-left to bottom-right: 1-4, 2-1, 4-1, 4-2, 5-1, 6-2, 6-4.
  • Figure 4: Ratio of solutions evaluated with and without GESP in the same optimization time. A higher value indicates that GESP was able to evaluate more solutions in the same time.
  • Figure 5: The objective value of the agents with respect to computation time in the mujoco tasks with and without GESP. Environments from top-left to bottom-right: ant, inverted double pendulum, swimmer, half cheetah, hopper, walker2d.
  • ...and 9 more figures

Theorems & Definitions (1)

  • Definition 1