Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation

TL;DR

This paper shows that deep RL policies learned from pixel inputs poorly generalize to small visual changes and that standard fine-tuning transfers often underperform or fail. It introduces a zero-shot transfer method using unaligned image-to-image translation (GANs) to map target visuals to a source domain, enabling the reuse of a source policy without retraining. When translated frames are imperfect, imitation learning from these demonstrations further accelerates learning and improves robustness. The approach yields substantial sample-efficiency gains on Breakout variants and Road Fighter levels, while also offering a practical framework for evaluating unaligned GANs in downstream RL tasks.

Abstract

Despite the remarkable success of Deep RL in learning control policies from raw pixels, the resulting models do not generalize. We demonstrate that a trained agent fails completely when facing small visual changes, and that fine-tuning---the common transfer learning paradigm---fails to adapt to these changes, to the extent that it is faster to re-train the model from scratch. We show that by separating the visual transfer task from the control policy we achieve substantially better sample efficiency and transfer behavior, allowing an agent trained on the source task to transfer well to the target tasks. The visual mapping from the target to the source domain is performed using unaligned GANs, resulting in a control policy that can be further improved using imitation learning from imperfect demonstrations. We demonstrate the approach on synthetic visual variants of the Breakout game, as well as on transfer between subsequent levels of Road Fighter, a Nintendo car-driving game. A visualization of our approach can be seen in https://youtu.be/4mnkzYyXMn4 and https://youtu.be/KCGTrQi6Ogo .

Figures (6)

• Figure 1: Various variations of the Breakout game: (a) Standard version, (b) A Constant Rectangle - a rectangle in the same size as the bricks is added to the background in a predefined location, (c) A Moving Square - a square is added to the background and its location changes to one of three predefined locations every $1000$ steps, (d) Green Lines - green lines in different sizes are drawn in the background, (e) Diagonals - diagonals are drawn in the left side of the background.
• Figure 2: A comparison between the different transfer techniques on Breakout. The y-axis shows the average reward per episode of Breakout during training. The x-axis shows the total number of training epochs where an epoch corresponds to 1 million frames. The plots are averaged on 3 runs with different random seeds. Each curve is the average and its background is the standard deviation.
• Figure 3: Illustration of a frame taken from the target task (left) and its matching frame of the source task generated with GANs (right) for each one of the Breakout variations. (a)-(d) demonstrate successes, while (e) and (f) show failure modes of the unaligned GAN. In (e) the ball in the input image is not generated in the output and in (f) not all bricks are generated, and some of the generated bricks appear smudged.
• Figure 4: Road Fighter levels from left to right: Level 1, Level 2, Level 3 and Level 4.
• Figure 5: Left: the original frame. Right: GAN generated. The first three examples show the success cases of the GAN while the last three show representative failures: in (d) and (f) the only object generated on the road is the player's car and in (e) the diagonal shaped road of level 2 in matched to the starting point of level 1.