Target Return Optimizer for Multi-Game Decision Transformer

TL;DR

MTRO introduces a data-driven approach to automatically select per-game target returns for the Multi-Game DT offline RL framework. It comprises DERD, which derives game-specific expert return distributions from offline episodes, and BARP, which adjusts the return predictions based on predictive accuracy and offline return frequencies. Evaluations on Atari show MTRO improves the IQM of human-normalized scores over the baseline Multi-Game DT without additional training, highlighting reduced reliance on human priors. The work advances scalable, autonomous generalization across games by aligning target returns with game-specific reward structures.

Abstract

Achieving autonomous agents with robust generalization capabilities across diverse games and tasks remains one of the ultimate goals in AI research. Recent advancements in transformer-based offline reinforcement learning, exemplified by the MultiGame Decision Transformer [Lee et al., 2022], have shown remarkable performance across various games or tasks. However, these approaches depend heavily on human expertise, presenting substantial challenges for practical deployment, particularly in scenarios with limited prior game-specific knowledge. In this paper, we propose an algorithm called Multi-Game Target Return Optimizer (MTRO) to autonomously determine game-specific target returns within the Multi-Game Decision Transformer framework using solely offline datasets. MTRO addresses the existing limitations by automating the target return configuration process, leveraging environmental reward information extracted from offline datasets. Notably, MTRO does not require additional training, enabling seamless integration into existing Multi-Game Decision Transformer architectures. Our experimental evaluations on Atari games demonstrate that MTRO enhances the performance of RL policies across a wide array of games, underscoring its potential to advance the field of autonomous agent development.

Figures (3)

• Figure 1: This figure compares distributions $P(\text{expert}_{t} \mid R_{t} \ldots)$ derived from \ref{['eq:expert_probability']} and \ref{['eq:expert_offline']}. Left: $P(\text{expert}_{t} \mid R_{t}, \ldots)$ under the assumption of \ref{['eq:expert_probability']}, which increases exponentially from $R_{\text{low}}$ to $R_{\text{high}}$ for the quantized returns in the range [-20, 100]. Middle: $P_\text{offline}(\text{expert}_{t} \mid R_{t}, \ldots)$ based on \ref{['eq:expert_offline']} for the Amidar game, where the distribution is closer to a normal distribution rather than an exponential type. Right: $P_\text{offline}(\text{expert}_{t} \mid R_{t}, \ldots)$ based on \ref{['eq:expert_offline']} for the TimePilot game, showing a distribution closer to an exponential type, but the maximum return value is significantly smaller than that in the left figure.
• Figure 2: This figure illustrates the differences between $N(R_1)$ derived from offline datasets, and the transformer's output $P(R_{t} \mid \ldots)$. Left: In the Breakout Game, $P(R_{t} \mid \ldots)$ tends to lean towards lower $R$ values compared to $N(R_t)$, suggesting that lower returns are more probable than what the offline datasets indicates. Right: In the Amidar Game, while $N(R_t)$ peaks at higher $R$ values, $P(R_{t} \mid \ldots)$ shows a notably different pattern, highlighting a divergence from the expected distribution.
• Figure 3: This figure demonstrates the effectiveness of target returns in MTRO by comparing the frequency of episode returns from offline datasets, $N(R_1)$, with $P(R_{t}, \ldots \mid \text{expert}_{t})$ used for sampling target returns in both Multi-Game DT and MTRO. Here, $P(R_{t}, \ldots \mid \text{expert}_{t})$ represents the distribution at the beginning of episodes in the games. In both games, it is evident that Multi-Game DT might select target returns in the range $R = 60 \sim 100$, which are not included in $N(R_1)$. On the other hand, because MTRO samples feasible target returns based on offline datasets, it is suitable as input to the transformer and is expected to generate expert actions.