Learning Human-Like RL Agents Through Trajectory Optimization With Action Quantization

TL;DR

This work reframes human-like reinforcement learning as trajectory optimization guided by human demonstrations. It introduces Macro Action Quantization (MAQ), which distills offline human data into a discrete codebook of macro actions via a Conditional-VQVAE and integrates this with receding-horizon control to constrain planning to human-like segments. By replacing primitive actions with macro-action indices, MAQ enables efficient long-horizon planning and demonstrably improves trajectory similarity and perceived human-likeness across multiple Adroit tasks and RL algorithms, including through a human Turing-like evaluation. The results suggest MAQ’s potential to produce more natural, trustworthy AI agents while remaining broadly compatible with existing RL methods. In short, MAQ provides a practical pathway to human-like behavior in RL by linking offline human patterns to online planning through a compact, discrete action representation.

Abstract

Human-like agents have long been one of the goals in pursuing artificial intelligence. Although reinforcement learning (RL) has achieved superhuman performance in many domains, relatively little attention has been focused on designing human-like RL agents. As a result, many reward-driven RL agents often exhibit unnatural behaviors compared to humans, raising concerns for both interpretability and trustworthiness. To achieve human-like behavior in RL, this paper first formulates human-likeness as trajectory optimization, where the objective is to find an action sequence that closely aligns with human behavior while also maximizing rewards, and adapts the classic receding-horizon control to human-like learning as a tractable and efficient implementation. To achieve this, we introduce Macro Action Quantization (MAQ), a human-like RL framework that distills human demonstrations into macro actions via Vector-Quantized VAE. Experiments on D4RL Adroit benchmarks show that MAQ significantly improves human-likeness, increasing trajectory similarity scores, and achieving the highest human-likeness rankings among all RL agents in the human evaluation study. Our results also demonstrate that MAQ can be easily integrated into various off-the-shelf RL algorithms, opening a promising direction for learning human-like RL agents. Our code is available at https://rlg.iis.sinica.edu.tw/papers/MAQ.

Figures (13)

• Figure 1: The overall training process of MAQ. (1) Human behavior distillation: A Conditional-VQVAE is trained on the state ($s_t$) and the macro action ($m_t=(a_t,a_{t+1},\ldots,a_{t+H-1})$) to learn a discrete codebook; the macro actions ($m_t$) are extracted from human demonstrations via a sliding window over action trajectories. (2) Reinforcement learning with MAQ: An online policy ($\pi_{\theta}$) acts in the learned discrete code space by selecting codebook indices, which the VQVAE decodes into macro action executed in the environment.
• Figure 2: Trajectory similarity scores and success rates with different lengths in MAQ+RLPD.
• Figure 3: Turing Test. Error bars represent 96% confidence intervals.
• Figure 4: Human-likeness ranking test.
• Figure 5: Agent behavior in Door.