Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Interpreting Behaviors and Geometric Constraints as Knowledge Graphs for Robot Manipulation Control

Chen Jiang, Allie Wang, Martin Jagersand

TL;DR

This work tackles the challenge of explainable robot manipulation by unifying high-level task scripting with low-level geometric constraints through event-centric robot knowledge graphs. By mapping behavior trees and geometric constraints into a cohesive graph-based representation, the approach enables semantic reasoning about actions and contexts while driving uncalibrated visual servoing control. The authors introduce a formal KG schema, define how behavior trees and four geometric constraint types are represented as semantic events, and provide a three-stage control loop that queries, conceptualizes, and enacts manipulation tasks, updating knowledge after completion. Real-world experiments demonstrate robustness, concurrency benefits, and task compositionality, while also highlighting limitations and opportunities for system scalability and automatic KG generation.

Abstract

In this paper, we investigate the feasibility of using knowledge graphs to interpret actions and behaviors for robot manipulation control. Equipped with an uncalibrated visual servoing controller, we propose to use robot knowledge graphs to unify behavior trees and geometric constraints, conceptualizing robot manipulation control as semantic events. The robot knowledge graphs not only preserve the advantages of behavior trees in scripting actions and behaviors, but also offer additional benefits of mapping natural interactions between concepts and events, which enable knowledgeable explanations of the manipulation contexts. Through real-world evaluations, we demonstrate the flexibility of the robot knowledge graphs to support explainable robot manipulation control.

Interpreting Behaviors and Geometric Constraints as Knowledge Graphs for Robot Manipulation Control

TL;DR

This work tackles the challenge of explainable robot manipulation by unifying high-level task scripting with low-level geometric constraints through event-centric robot knowledge graphs. By mapping behavior trees and geometric constraints into a cohesive graph-based representation, the approach enables semantic reasoning about actions and contexts while driving uncalibrated visual servoing control. The authors introduce a formal KG schema, define how behavior trees and four geometric constraint types are represented as semantic events, and provide a three-stage control loop that queries, conceptualizes, and enacts manipulation tasks, updating knowledge after completion. Real-world experiments demonstrate robustness, concurrency benefits, and task compositionality, while also highlighting limitations and opportunities for system scalability and automatic KG generation.

Abstract

In this paper, we investigate the feasibility of using knowledge graphs to interpret actions and behaviors for robot manipulation control. Equipped with an uncalibrated visual servoing controller, we propose to use robot knowledge graphs to unify behavior trees and geometric constraints, conceptualizing robot manipulation control as semantic events. The robot knowledge graphs not only preserve the advantages of behavior trees in scripting actions and behaviors, but also offer additional benefits of mapping natural interactions between concepts and events, which enable knowledgeable explanations of the manipulation contexts. Through real-world evaluations, we demonstrate the flexibility of the robot knowledge graphs to support explainable robot manipulation control.
Paper Structure (15 sections, 8 equations, 3 figures, 3 tables, 1 algorithm)

This paper contains 15 sections, 8 equations, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Knowledge Graphs and Behavior Trees in Robotics
  4. Geometric Constraints in Robotics
  5. Robot Knowledge Graphs
  6. Structure of Robot Knowledge Graphs
  7. Behavior Trees in Knowledge Graphs
  8. Geometric Constraints as Behaviors
  9. Control with Robot Knowledge Graphs
  10. Experimental Evaluations
  11. Evaluation Settings
  12. Results on Correctness of Constraints
  13. Results on Concurrent Constraints
  14. Results on Real-world Robot Control
  15. Conclusions

Figures (3)

  • Figure 1: An example of a robot knowledge graph to conceptualize a behavior tree and geometric constraints (a point-to-point and a line-to-line constraint) for a carrot grasping task.
  • Figure 2: The schema for robot event knowledge graphs. The hollow arrow marks owl:subClassOf relation, while the regular arrow marks dedicated relations.
  • Figure 3: Visualization of robot knowledge graphs to conceptualize pick-and-place tasks: (a) lemon $\rightarrow$ bowl; (b) marker pen $\rightarrow$ box. Behaviors and robot events are marked in purple, contexts in blue, and linked entities to DBpedia in green.