Acquiring Grounded Representations of Words with Situated Interactive Instruction

TL;DR

The paper tackles grounding word meanings across adjectives, nouns, prepositions, and verbs by enabling mixed-initiative, situated instruction with a human mentor. It implements the approach in the Soar cognitive architecture and demonstrates online, incremental learning on a table-top robot arm, integrating perception-derived features with semantic, episodic, and procedural knowledge. The method grounds words to perceptual symbols, spatial relations, and action concepts, enabling the agent to learn quickly from few examples and to initiate learning when needed. The study shows flexible human-agent collaboration, efficient acquisition across word classes, and potential for scalable grounding in real-world robotic manipulation tasks.

Abstract

We present an approach for acquiring grounded representations of words from mixed-initiative, situated interactions with a human instructor. The work focuses on the acquisition of diverse types of knowledge including perceptual, semantic, and procedural knowledge along with learning grounded meanings. Interactive learning allows the agent to control its learning by requesting instructions about unknown concepts, making learning efficient. Our approach has been instantiated in Soar and has been evaluated on a table-top robotic arm capable of manipulating small objects.

Figures (6)

• Figure 1: System overview including major Soar components.
• Figure 2: Phases in the interaction cycle.
• Figure 3: Annotated human-agent dialog for acquisition of store
• Figure 4: Top down view of objects and the representation of left of learned from the three marked examples.
• Figure 5: (Left) Pre-encoded linguistic map for put. (Right) Acquired action-concept network for store.