Boosting Deep Reinforcement Learning with Semantic Knowledge for Robotic Manipulators

TL;DR

This work tackles the sample inefficiency of deep reinforcement learning (DRL) in robotic manipulation by injecting semantic knowledge through Knowledge Graph Embeddings (KGEs). A subgraph-based extraction combined with GloVe embeddings is concatenated with visual features and integrated into an A3C-based DRL architecture, before the LSTM policy module. Empirical results across TIAGo and IRB120 robots show that KGEs reduce learning time by up to $60\%$ and improve accuracy by up to 20 percentage points, with further gains when domain randomization (DR) is applied to target attributes. The approach achieves these improvements with minimal additional computational cost and demonstrates the potential of semantic knowledge to enhance DRL performance in robotics, including future sim-to-real transfer work and exploration of alternative integration points.

Abstract

Deep Reinforcement Learning (DRL) is a powerful framework for solving complex sequential decision-making problems, particularly in robotic control. However, its practical deployment is often hindered by the substantial amount of experience required for learning, which results in high computational and time costs. In this work, we propose a novel integration of DRL with semantic knowledge in the form of Knowledge Graph Embeddings (KGEs), aiming to enhance learning efficiency by providing contextual information to the agent. Our architecture combines KGEs with visual observations, enabling the agent to exploit environmental knowledge during training. Experimental validation with robotic manipulators in environments featuring both fixed and randomized target attributes demonstrates that our method achieves up to {60}{\%} reduction in learning time and improves task accuracy by approximately 15 percentage points, without increasing training time or computational complexity. These results highlight the potential of semantic knowledge to reduce sample complexity and improve the effectiveness of DRL in robotic applications.

Figures (11)

• Figure 1: Proposed framework diagram. A subgraph selector is used to extract the important entities and relationships based on the environment from a complete graph. This subgraph is embedded and concatenated in the layer prior to the policy approximation blocks of the DRL agent architecture. The result is an improvement in learning times and accuracy and a reduction of the required exploration.
• Figure 2: Environment observation with the three targets for the TIAGo and IRB120 scenarios. Note that we have added a black sphere on each grasping point for convenience. During training, the resolution is 64$\times$64, and the sphere does not appear. The site axes correspond to the $x$ coordinate in red, the $y$ coordinate in green, and the $z$ coordinate in blue.
• Figure 3: Proposed A3C architecture to integrate KGEs in the DRL agent's learning process. The KGE size depends on the number of features encoded. The output is divided into seven actors for the TIAGo and six for the IRB120, plus the critic value.
• Figure 4: Training curves of the agents trained with the TIAGo and fixed targets' colors. The blue curve is the full KGE model, the green is the partial KGE, and the orange is the BM. The training process lasts for 70 M steps.
• Figure 5: Training curves of the agents trained with the IRB120 and fixed targets' colors. The blue curve is the full KGE model, the green is the partial KGE, and the orange is the BM. The training process lasts for 70 M steps.