Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Using The Concept Hierarchy for Household Action Recognition

Andrei Costinescu, Luis Figueredo, Darius Burschka

Abstract

We propose a method to systematically represent both the static and the dynamic components of environments, i.e. objects and agents, as well as the changes that are happening in the environment, i.e. the actions and skills performed by agents. Our approach, the Concept Hierarchy, provides the necessary information for autonomous systems to represent environment states, perform action modeling and recognition, and plan the execution of tasks. Additionally, the hierarchical structure supports generalization and knowledge transfer to environments. We rigorously define tasks, actions, skills, and affordances that enable human-understandable action and skill recognition.

Using The Concept Hierarchy for Household Action Recognition

Abstract

We propose a method to systematically represent both the static and the dynamic components of environments, i.e. objects and agents, as well as the changes that are happening in the environment, i.e. the actions and skills performed by agents. Our approach, the Concept Hierarchy, provides the necessary information for autonomous systems to represent environment states, perform action modeling and recognition, and plan the execution of tasks. Additionally, the hierarchical structure supports generalization and knowledge transfer to environments. We rigorously define tasks, actions, skills, and affordances that enable human-understandable action and skill recognition.
Paper Structure (13 sections, 5 figures)

This paper contains 13 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Contributions
  3. Structure
  4. The Content of the Concept Hierarchy
  5. Concepts, Instances and ValueDomains: Knowledge Definitions, Containers, and Data Types
  6. Functions: Mathematical Modifiers of ValueDomains
  7. Actions: Modifiers of Knowledge Containers
  8. Skills: Action Implementation in Environments
  9. Affordances
  10. Motion Primitives, Abilities: Building Blocks of Skills
  11. Tasks: Multiple Changes in the Environment
  12. Action and Skill Recognition
  13. Conclusion

Figures (5)

  • Figure 1: "How to transform the left environment into the right one?" The knowledge in the Concept Hierarchy enables household robots to represent environments and to create a plan to execute tasks.
  • Figure 2: The task of transforming the left environment to the desired right one is divided into actions that are correlated to skills, which are composed of Motion Primitives (MP). * marks knowledge from the CH: action and skill definitions, skill-to-action association, sequencing of motion primitives for skill execution, and entity properties.
  • Figure 3: The CH contains Objects, Surfaces, Agents, and Grippers. Their specific properties are modeled as ValueDomains. Skills and Actions contain ValueDomain parameters, including Objects and Agents. Skills perform Actions in an environment. They have requirements from and effects on their parameters and can check whether a skill is active. These are modeled as Functions with ValueDomain arguments.
  • Figure 4: Skill recognition result of a bimanual task of pouring milk into a bowl. The demonstration is composed of closing and then opening a milk box with the left hand, pouring milk into the bowl, and closing the milk box with the right hand. The upper figure presents the Skill instances our method recognizes for each hand and Skill type. The colors help distinguish Skills of the same type with different parameters. The lower figure shows the results of hao_pgcn that was trained on the Bimanual Actions dataset bimacs_dataset (darker confidence = higher softmax output).
  • Figure 5: The procedure for Skill recognition.