Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Language Model-Based Paired Variational Autoencoders for Robotic Language Learning

Ozan Özdemir, Matthias Kerzel, Cornelius Weber, Jae Hee Lee, Stefan Wermter

TL;DR

The paper addresses embodied language grounding by linking robot actions to descriptive language within a tabletop manipulation task. It proposes PVAE, with two VAEs bound via a binding loss, and PVAE-BERT, which replaces the language encoder with a pretrained BERT model to handle unconstrained natural language; a channel-separated CAE improves visual feature discrimination, especially color. The key contributions are (i) demonstrating one-to-many action–description binding with variational latent spaces, (ii) showing robust translation across color/shapes and action directions, and (iii) enabling scalable language understanding through BERT finetuning for real-world human instructions. The findings suggest that PVAE variants outperform prior PRAE-based approaches, and that integrating a pretrained language model enables practical, unconstrained language collaboration with robots, indicating strong potential for real-world deployment and interaction with human users.

Abstract

Human infants learn language while interacting with their environment in which their caregivers may describe the objects and actions they perform. Similar to human infants, artificial agents can learn language while interacting with their environment. In this work, first, we present a neural model that bidirectionally binds robot actions and their language descriptions in a simple object manipulation scenario. Building on our previous Paired Variational Autoencoders (PVAE) model, we demonstrate the superiority of the variational autoencoder over standard autoencoders by experimenting with cubes of different colours, and by enabling the production of alternative vocabularies. Additional experiments show that the model's channel-separated visual feature extraction module can cope with objects of different shapes. Next, we introduce PVAE-BERT, which equips the model with a pretrained large-scale language model, i.e., Bidirectional Encoder Representations from Transformers (BERT), enabling the model to go beyond comprehending only the predefined descriptions that the network has been trained on; the recognition of action descriptions generalises to unconstrained natural language as the model becomes capable of understanding unlimited variations of the same descriptions. Our experiments suggest that using a pretrained language model as the language encoder allows our approach to scale up for real-world scenarios with instructions from human users.

Language Model-Based Paired Variational Autoencoders for Robotic Language Learning

TL;DR

The paper addresses embodied language grounding by linking robot actions to descriptive language within a tabletop manipulation task. It proposes PVAE, with two VAEs bound via a binding loss, and PVAE-BERT, which replaces the language encoder with a pretrained BERT model to handle unconstrained natural language; a channel-separated CAE improves visual feature discrimination, especially color. The key contributions are (i) demonstrating one-to-many action–description binding with variational latent spaces, (ii) showing robust translation across color/shapes and action directions, and (iii) enabling scalable language understanding through BERT finetuning for real-world human instructions. The findings suggest that PVAE variants outperform prior PRAE-based approaches, and that integrating a pretrained language model enables practical, unconstrained language collaboration with robots, indicating strong potential for real-world deployment and interaction with human users.

Abstract

Human infants learn language while interacting with their environment in which their caregivers may describe the objects and actions they perform. Similar to human infants, artificial agents can learn language while interacting with their environment. In this work, first, we present a neural model that bidirectionally binds robot actions and their language descriptions in a simple object manipulation scenario. Building on our previous Paired Variational Autoencoders (PVAE) model, we demonstrate the superiority of the variational autoencoder over standard autoencoders by experimenting with cubes of different colours, and by enabling the production of alternative vocabularies. Additional experiments show that the model's channel-separated visual feature extraction module can cope with objects of different shapes. Next, we introduce PVAE-BERT, which equips the model with a pretrained large-scale language model, i.e., Bidirectional Encoder Representations from Transformers (BERT), enabling the model to go beyond comprehending only the predefined descriptions that the network has been trained on; the recognition of action descriptions generalises to unconstrained natural language as the model becomes capable of understanding unlimited variations of the same descriptions. Our experiments suggest that using a pretrained language model as the language encoder allows our approach to scale up for real-world scenarios with instructions from human users.
Paper Structure (19 sections, 8 equations, 4 figures, 5 tables)

This paper contains 19 sections, 8 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Language-to-Action Translation
  4. Action-to-Language Translation
  5. Bidirectional Translation
  6. Proposed Methods: PVAE & PVAE-BERT
  7. Language Variational Autoencoder
  8. Action Variational Autoencoder
  9. Visual Feature Extraction
  10. Sampling and Binding
  11. Loss Function
  12. Transformer-Based Language Encoder
  13. Training Details
  14. Evaluation and Results
  15. Experiment 1
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Our tabletop object manipulation scenario in the simulation environment: the NICO robot is interacting with toy objects. In the left panel, NICO views all the toy objects; on the right, NICO pulls the red house. In both panels, NICO's field of view is given in the top right inset.
  • Figure 2: The architecture of the proposed PVAE and PVAE-BERT models: the language VAE (blue rectangles) processes descriptions, whilst the action VAE (orange rectangles) processes joint angles and images at each time step. The input to the language VAE is the given description $x$, whereas the action VAE takes as input joint angle values $j$ and visual features $v$. The two VAEs are implicitly bound via a binding loss in the latent representation space. The image from which the v1 is extracted is magnified for visualisation purposes. $<$BOS$>$ and $<$EOS$>$ stand for beginning of sentence and end of sentence tags, respectively. The two models differ only by the language encoder employed: the PVAE uses LSTM, whereas PVAE-BERT uses a pretrained BERT model.
  • Figure 3: Examples of language-to-action translation by PVAE-BERT and its comparison with PVAE: in the top row, the two plots represent the ground truth and predicted joint trajectories by PVAE-BERT for PUSH-LEFT-SLOWLY and PULL-LEFT-SLOWLY actions. Solid lines show the ground truth, while the dashed lines, which are often covered by the solid lines, show the predicted joint angle values. In the bottom row, the left plot shows the total error margin of the five joint values produced by PVAE and PVAE-BERT per time step for the PUSH-LEFT-SLOWLY action, while the right plot shows the joint values produced by PVAE-BERT given three variations (see Table \ref{['tab:descvar']}) of the same command for PULL-LEFT-SLOWLY - notice how the joint trajectories overlap most of the time. In all of the plots, the X axis represents the time steps.
  • Figure 4: Hidden features of language (a) and hidden features of action (b): PCA was performed jointly on the hidden features of 36 descriptions and the hidden features of 144 actions. For (b), each unique action (12 in total) occurs 12 times as there are 12 possible cube arrangements; therefore, 144 points are shown. For both (a) and (b), we label the points according to descriptions, i.e., for (b), actions are also labelled according to their paired descriptions. As can be seen from the legend, different shapes, colours and fillings indicate the verb (action type), object colour and adverb (speed), respectively.