Ask-AC: An Initiative Advisor-in-the-Loop Actor-Critic Framework

TL;DR

Ask-AC tackles the inefficiency of passive advisor supervision in reinforcement learning by introducing a bidirectional, initiative-based framework that enables the agent to ask for advice on demand. The method combines an action requester with an adaptive state selector, which together determine when to seek advisor input and adapt to environment changes, and it can be plugged into various discrete actor-critic backbones. Empirical results across stationary and non-stationary tasks show that Ask-AC substantially improves learning efficiency, achieving performance comparable to continuous advisor monitoring while reducing advisor burden, and it demonstrates robustness to imperfect advisor feedback and favorable outcomes with human advisors. The work holds practical value for scalable, on-demand human-in-the-loop learning in dynamic environments and opens avenues for extending the approach to real-time control and human preferences.

Abstract

Despite the promising results achieved, state-of-the-art interactive reinforcement learning schemes rely on passively receiving supervision signals from advisor experts, in the form of either continuous monitoring or pre-defined rules, which inevitably result in a cumbersome and expensive learning process. In this paper, we introduce a novel initiative advisor-in-the-loop actor-critic framework, termed as Ask-AC, that replaces the unilateral advisor-guidance mechanism with a bidirectional learner-initiative one, and thereby enables a customized and efficacious message exchange between learner and advisor. At the heart of Ask-AC are two complementary components, namely action requester and adaptive state selector, that can be readily incorporated into various discrete actor-critic architectures. The former component allows the agent to initiatively seek advisor intervention in the presence of uncertain states, while the latter identifies the unstable states potentially missed by the former especially when environment changes, and then learns to promote the ask action on such states. Experimental results on both stationary and non-stationary environments and across different actor-critic backbones demonstrate that the proposed framework significantly improves the learning efficiency of the agent, and achieves the performances on par with those obtained by continuous advisor monitoring.

Figures (8)

• Figure 1: Comparing the proposed initiative Ask-AC framework with the existing passive framework for interactive reinforcement learning. Unlike the passive framework with unidirectional interactions, Ask-AC introduces a bidirectional message exchange scheme to endow the agent with an initiatively-asking mechanism, which largely reduces the demand for advisor feedback. "Env." denotes environment.
• Figure 2: Illustration of the proposed Ask-AC framework. Ask-AC is an initiative advisor-in-the-loop actor-critic framework that comprises two complementary components, namely action requester and adaptive state selector.
• Figure 3: Comparing the Ask-AC framework with and without the adaptive state selector in the non-stationary environment, where the environment model changes at the $1.0\times10^5$ step. The shaded region represents one standard deviation of the average evaluation over 5 trials.
• Figure 4: Learning curves for eight tasks in four stationary environments. We change the CartPole and DoorKey into six tasks by modifying the internal parameters to test the performance of our method over varying difficulty. For CartPole, we modify the length of the pole $L$ to 0.5, 1.0 and 2.0. For the DoorKey, we modify the size of the maze $S$ to $5\times5$, $6\times6$ and $8\times8$. The shaded region represents one standard deviation of the average evaluation over 5 trials.
• Figure 5: Learning curves for the Non-stationary CartPole environment and the Non-stationary DoorKey environment, trained for three hundred thousand timesteps. In the Non-stationary CartPole environment, we initialize the length of the pole $L$ to $0.5$, and then modify the length of the pole $L$ to $1.0$ and $2.0$ at the $1.0\times 10^5$ and $2.0\times 10^5$ step, respectively. In the Non-stationary DoorKey environment, we initialize the size of the maze $S$ to 5x5, and then modify the size of the maze $S$ to $6\times6$ and $8\times8$ at the $1.0\times 10^5$ and $2.0\times 10^5$ step, respectively.