Closed-loop Teaching via Demonstrations to Improve Policy Transparency

TL;DR

The paper addresses the problem that a priori demonstration curricula for policy transparency may misalign with a learner's evolving understanding. It proposes a closed-loop teaching framework that interleaves demonstrations, diagnostic tests, and feedback, guided by a Bayesian particle-filter model of human beliefs and governed by ZPD and the testing effect. Key contributions include the particle-filter human belief model, a closed-loop teaching pipeline, and empirical evidence from a user study showing a 43% reduction in test regret compared to a baseline, across delivery and skateboard domains. The work advances policy transparency in interactive RL by enabling in situ calibration of explanations to the human learner, potentially improving interpretability and trust in AI systems.

Abstract

Demonstrations are a powerful way of increasing the transparency of AI policies. Though informative demonstrations may be selected a priori through the machine teaching paradigm, student learning may deviate from the preselected curriculum in situ. This paper thus explores augmenting a curriculum with a closed-loop teaching framework inspired by principles from the education literature, such as the zone of proximal development and the testing effect. We utilize tests accordingly to close to the loop and maintain a novel particle filter model of human beliefs throughout the learning process, allowing us to provide demonstrations that are targeted to the human's current understanding in real time. A user study finds that our proposed closed-loop teaching framework reduces the regret in human test responses by 43% over a baseline.

Figures (7)

• Figure 1: (a) Previous works aim to improve policy transparency via a set of demonstrations selected a priori, but student learning may deviate from the expected trajectory. (b) We propose a closed-loop teaching framework using tests and feedback to detect and correct for such deviations in situ.
• Figure 2: Sample teaching sequence for a batch of KCs on mud cost. (a) First demonstration (green) contrasts with a counterfactual alternative likely considered by a human (orange), which conveys that mud is costly. (b) Second demonstration lowerbounds mud cost. (c) Human is asked to predict the robot's behavior in a test. (d) Incorrect response suggests that the demonstration was not understood. (e) Human is given the correct response as feedback. (f) Remedial demonstration is provided to target the misunderstanding. (g) Human is given a remedial test. (h) Correct answer suggests understanding.
• Figure 3: Example sequence on how a demonstration updates a particle filter model of human beliefs. The robot reward function is shown as a red dot, and the constraint consistent with the demonstration is shown in all plots for reference. (a) Particles before demonstration (prior). (b) Demonstration shown to the human, alongside a counterfactual that considers mud to be slightly negative or positive, or neutral. (c) The constraint (Eq. \ref{['eq:BEC_demo']}) consistent with the demonstration that conveys that mud must be at least twice as costly as an action, visualized with the uniform distribution portion of the custom distribution (Fig. \ref{['fig:uniform_vmf']}) used to update particle weights. (d) Particles after demonstration (posterior).
• Figure 4: Cross-section of the spherical probability density function used to update particle weights given a constraint generated from a demonstration (Eq. \ref{['eq:BEC_demo']}).
• Figure 5: Proposed closed-loop teaching framework. Knowledge components (KCs) are passed to the AI teacher as a lesson. The demonstrator generates demonstrations that convey the KCs, the tester provides test(s), and the evaluator analyzes the test response(s), provides feedback on its correctness, and updates the model of human knowledge. If the human fails to learn a KC through two rounds of demonstrations and tests, the switch (labeled 'S') flips such that only tests and feedback are provided until an understanding of the remaining KCs is demonstrated through correct responses.