Dynamic Observation Policies in Observation Cost-Sensitive Reinforcement Learning

TL;DR

The Deep Dynamic Multi-Step Observationless Agent (DMSOA) is proposed, compared with the literature and empirically evaluated on OpenAI gym and Atari Pong environments show that DMSOA learns a better policy with fewer decision steps and measurements than the considered alternative from the literature.

Abstract

Reinforcement learning (RL) has been shown to learn sophisticated control policies for complex tasks including games, robotics, heating and cooling systems and text generation. The action-perception cycle in RL, however, generally assumes that a measurement of the state of the environment is available at each time step without a cost. In applications such as materials design, deep-sea and planetary robot exploration and medicine, however, there can be a high cost associated with measuring, or even approximating, the state of the environment. In this paper, we survey the recently growing literature that adopts the perspective that an RL agent might not need, or even want, a costly measurement at each time step. Within this context, we propose the Deep Dynamic Multi-Step Observationless Agent (DMSOA), contrast it with the literature and empirically evaluate it on OpenAI gym and Atari Pong environments. Our results, show that DMSOA learns a better policy with fewer decision steps and measurements than the considered alternative from the literature.

Figures (6)

• Figure 1: Illustration of the DMSOA Framework.
• Figure 2: Mean and standard deviation of performance on Cartpole (upper) and Acrobot (lower). Left: costed reward and right: episode length. In both cases, DMSOA has a higher mean costed reward, and is superior in terms of the control objective (longer episodes on Cartpole and shorter episodes on Acrobot.)
• Figure 3: Measurement behaviour of the best OSMBOA (left) and DMSOA (right) policies. Top: Cartpole, centre: Acrobot, lower: Lunar Lander. For OSMBOA, blue indicates that no measurement was made and orange indicates that a measurement was made. In the case of DMSOA, blue indicates that a measurement is made after one step, orange indicates that a measurement is made after two steps, and red indicates that a measurement is made after three steps. This figure reveals the very distinct measurement behaviour between the two classes of AC-NOMDP agents.
• Figure 4: Mean and standard deviation of the costed reward on the Atari Pong environment. DMSOA learns a policy that for a better costed reward.
• Figure 5: Mean and standard deviation of the performance on the Lunar Lander environment. Top left: episode length, top right: sum of the number of successful landings, lower: costed reward. DMSOA and OSMBOA achieve similar mean costed rewards. DMSOA, however, learns to successfully land the ship more frequently and in fewer steps.