Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TeNet: Text-to-Network for Compact Policy Synthesis

Ariyan Bighashdel, Kevin Sebastian Luck

TL;DR

TeNet addresses the challenge of building language-enabled robot controllers that are both compact and suitable for real-time deployment. It introduces a language-conditioned hypernetwork that generates executable policy parameters from LLM-derived text embeddings, enabling control solely from low-dimensional state inputs at high frequencies. A grounded variant further aligns language with demonstrated behavior during training, improving generalization in multi-task and meta-learning settings. Empirical results on MuJoCo and Meta-World show TeNet achieves substantially smaller controllers (≈$40{,}000$ parameters) and high control rates (>$9\ \mathrm{kHz}$), while outperforming trajectory-prompt baselines in diverse tasks, though it remains data-hungry and benefits from richer language models for robustness to paraphrase. Overall, TeNet demonstrates a practical path to scalable, language-driven control in resource-constrained robotic systems, with groundable extensions promising broader applicability including perception and reinforcement fine-tuning.

Abstract

Robots that follow natural-language instructions often either plan at a high level using hand-designed interfaces or rely on large end-to-end models that are difficult to deploy for real-time control. We propose TeNet (Text-to-Network), a framework for instantiating compact, task-specific robot policies directly from natural language descriptions. TeNet conditions a hypernetwork on text embeddings produced by a pretrained large language model (LLM) to generate a fully executable policy, which then operates solely on low-dimensional state inputs at high control frequencies. By using the language only once at the policy instantiation time, TeNet inherits the general knowledge and paraphrasing robustness of pretrained LLMs while remaining lightweight and efficient at execution time. To improve generalization, we optionally ground language in behavior during training by aligning text embeddings with demonstrated actions, while requiring no demonstrations at inference time. Experiments on MuJoCo and Meta-World benchmarks show that TeNet produces policies that are orders of magnitude smaller than sequence-based baselines, while achieving strong performance in both multi-task and meta-learning settings and supporting high-frequency control. These results show that text-conditioned hypernetworks offer a practical way to build compact, language-driven controllers for ressource-constrained robot control tasks with real-time requirements.

TeNet: Text-to-Network for Compact Policy Synthesis

TL;DR

TeNet addresses the challenge of building language-enabled robot controllers that are both compact and suitable for real-time deployment. It introduces a language-conditioned hypernetwork that generates executable policy parameters from LLM-derived text embeddings, enabling control solely from low-dimensional state inputs at high frequencies. A grounded variant further aligns language with demonstrated behavior during training, improving generalization in multi-task and meta-learning settings. Empirical results on MuJoCo and Meta-World show TeNet achieves substantially smaller controllers (≈ parameters) and high control rates (>), while outperforming trajectory-prompt baselines in diverse tasks, though it remains data-hungry and benefits from richer language models for robustness to paraphrase. Overall, TeNet demonstrates a practical path to scalable, language-driven control in resource-constrained robotic systems, with groundable extensions promising broader applicability including perception and reinforcement fine-tuning.

Abstract

Robots that follow natural-language instructions often either plan at a high level using hand-designed interfaces or rely on large end-to-end models that are difficult to deploy for real-time control. We propose TeNet (Text-to-Network), a framework for instantiating compact, task-specific robot policies directly from natural language descriptions. TeNet conditions a hypernetwork on text embeddings produced by a pretrained large language model (LLM) to generate a fully executable policy, which then operates solely on low-dimensional state inputs at high control frequencies. By using the language only once at the policy instantiation time, TeNet inherits the general knowledge and paraphrasing robustness of pretrained LLMs while remaining lightweight and efficient at execution time. To improve generalization, we optionally ground language in behavior during training by aligning text embeddings with demonstrated actions, while requiring no demonstrations at inference time. Experiments on MuJoCo and Meta-World benchmarks show that TeNet produces policies that are orders of magnitude smaller than sequence-based baselines, while achieving strong performance in both multi-task and meta-learning settings and supporting high-frequency control. These results show that text-conditioned hypernetworks offer a practical way to build compact, language-driven controllers for ressource-constrained robot control tasks with real-time requirements.
Paper Structure (25 sections, 8 equations, 4 figures, 2 tables)

This paper contains 25 sections, 8 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Statement
  4. Method
  5. Direct TeNet
  6. Grounded TeNet
  7. Direct alignment (MSE).
  8. Contrastive alignment.
  9. Summary.
  10. Experiments
  11. Experimental Setup
  12. Benchmarks.
  13. Models.
  14. Metrics & protocol.
  15. Defaults.
  16. ...and 10 more sections

Figures (4)

  • Figure 1: Training (top) and inference (bottom) of the proposed framework. During training, trajectories and task descriptions are encoded, projected, and aligned through a language grounding module, with a hypernetwork generating task-specific policies optimized by imitation and grounding losses. At inference, only the task description conditions the hypernetwork to instantiate a policy that maps states to actions.
  • Figure 2: Performance across Mujoco (HalfCheetah-Dir, HalfCheetah-Vel, Ant-Dir) and Meta-World (ML1 Pick-Place, MT10, MT50). Each subplot reports mean and standard deviation over three seeds. A shared legend is shown at the top.
  • Figure 3: TeNet-Contrast performance on ML1 Pick-Place with varying numbers of tasks.
  • Figure 4: Achieved forward velocity vs. instructed target velocity in HalfCheetah-Vel (mean over 50 rollouts).