Privacy Risks in Reinforcement Learning for Household Robots

TL;DR

This work shows that privacy leakage is a real risk in reinforcement learning for embodied household robots by presenting gradient-inversion attacks that reconstruct private inputs from training gradients. It introduces two attack frameworks, Deep Q-learning Gradient Inversion (QGI) and REINFORCE Gradient Inversion (RGI), which reconstruct multimodal state information, actions, and supervision signals using an objective-agnostic gradient calculation and a combination of optimization- and rule-based steps. The methods are validated in the AI2THOR active-perception environment, achieving high-fidelity reconstructions of RGB/depth images and vector coordinates, near-perfect action recovery, and sub-percent errors in Q-values and returns. These results underscore substantial privacy risks in gradient-sharing scenarios and motivate the development of privacy-preserving mechanisms for embodied RL systems.

Abstract

The prominence of embodied Artificial Intelligence (AI), which empowers robots to navigate, perceive, and engage within virtual environments, has attracted significant attention, owing to the remarkable advances in computer vision and large language models. Privacy emerges as a pivotal concern within the realm of embodied AI, as the robot accesses substantial personal information. However, the issue of privacy leakage in embodied AI tasks, particularly concerning reinforcement learning algorithms, has not received adequate consideration in research. This paper aims to address this gap by proposing an attack on the training process of the value-based algorithm and the gradient-based algorithm, utilizing gradient inversion to reconstruct states, actions, and supervisory signals. The choice of using gradients for the attack is motivated by the fact that commonly employed federated learning techniques solely utilize gradients computed based on private user data to optimize models, without storing or transmitting the data to public servers. Nevertheless, these gradients contain sufficient information to potentially expose private data. To validate our approach, we conducted experiments on the AI2THOR simulator and evaluated our algorithm on active perception, a prevalent task in embodied AI. The experimental results demonstrate the effectiveness of our method in successfully reconstructing all information from the data in 120 room layouts. Check our website for videos.

Figures (4)

• Figure 1: Our method (Attacker) can reconstruct the images of your untidy room and window orientation from the gradient.
• Figure 2: Pipeline of the proposed gradient inversion attack on DQN (left) and REINFORCE (right). The left part of each image shows the steps to reconstruct the multimodal information, and the right part explains each step in detail.
• Figure 3: Qualitative results. The left shows the results of depth+RGB images (S1). The right shows the results of depth images (S2). The upper shows the results of QGI. The lower shows the results of RGI.
• Figure 4: Ablation study of multimodal state reconstruction. Comparison of joint optimization (joint) and our method.