Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Weakly-supervised Latent Models for Task-specific Visual-Language Control

Xian Yeow Lee, Lasitha Vidyaratne, Gregory Sin, Ahmed Farahat, Chetan Gupta

TL;DR

The paper tackles the problem of spatially grounded control for autonomous visual inspection by showing that naive multimodal LLM planners struggle to center objects in images. It introduces a task-specific latent dynamics model that learns action-conditioned latent shifts $\Delta_\theta(z_s,z_a)$ under weak supervision from goal-state prototypes $z^*$, using global action embeddings $g_a$ to stabilize learning. The approach achieves about $70$–$71\%$ accuracy on held-out images and instructions, significantly outperforming zero-shot LLM planners and demonstrating robust generalization to unseen visuals and language. This work suggests that compact, domain-specific latent representations can efficiently enable precise spatial alignment, offering a practical tool to augment agentic planning in hazardous environments with safety and scalability benefits.

Abstract

Autonomous inspection in hazardous environments requires AI agents that can interpret high-level goals and execute precise control. A key capability for such agents is spatial grounding, for example when a drone must center a detected object in its camera view to enable reliable inspection. While large language models provide a natural interface for specifying goals, using them directly for visual control achieves only 58\% success in this task. We envision that equipping agents with a world model as a tool would allow them to roll out candidate actions and perform better in spatially grounded settings, but conventional world models are data and compute intensive. To address this, we propose a task-specific latent dynamics model that learns state-specific action-induced shifts in a shared latent space using only goal-state supervision. The model leverages global action embeddings and complementary training losses to stabilize learning. In experiments, our approach achieves 71\% success and generalizes to unseen images and instructions, highlighting the potential of compact, domain-specific latent dynamics models for spatial alignment in autonomous inspection.

Weakly-supervised Latent Models for Task-specific Visual-Language Control

TL;DR

The paper tackles the problem of spatially grounded control for autonomous visual inspection by showing that naive multimodal LLM planners struggle to center objects in images. It introduces a task-specific latent dynamics model that learns action-conditioned latent shifts under weak supervision from goal-state prototypes , using global action embeddings to stabilize learning. The approach achieves about – accuracy on held-out images and instructions, significantly outperforming zero-shot LLM planners and demonstrating robust generalization to unseen visuals and language. This work suggests that compact, domain-specific latent representations can efficiently enable precise spatial alignment, offering a practical tool to augment agentic planning in hazardous environments with safety and scalability benefits.

Abstract

Autonomous inspection in hazardous environments requires AI agents that can interpret high-level goals and execute precise control. A key capability for such agents is spatial grounding, for example when a drone must center a detected object in its camera view to enable reliable inspection. While large language models provide a natural interface for specifying goals, using them directly for visual control achieves only 58\% success in this task. We envision that equipping agents with a world model as a tool would allow them to roll out candidate actions and perform better in spatially grounded settings, but conventional world models are data and compute intensive. To address this, we propose a task-specific latent dynamics model that learns state-specific action-induced shifts in a shared latent space using only goal-state supervision. The model leverages global action embeddings and complementary training losses to stabilize learning. In experiments, our approach achieves 71\% success and generalizes to unseen images and instructions, highlighting the potential of compact, domain-specific latent dynamics models for spatial alignment in autonomous inspection.

Paper Structure

This paper contains 19 sections, 7 equations, 4 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Problem Formulation and Method
  4. Problem Definition
  5. Latent Dynamics Model
  6. Training Objectives
  7. Model architectures and training
  8. Results
  9. Multi-modal LLM as Planners
  10. Latent Dynamics Model as a Planner
  11. Ablation Studies
  12. Analysis of failures
  13. Benefits and Limitations
  14. Conclusion
  15. Broader Impacts
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Overview of an LLM-powered AI agent leveraging tools, including a latent dynamics model, for improved spatially grounded visual-language control in inspection tasks.
  • Figure 2: Example images of a pressure gauge. (a) Raw images, (b) Images with bounding box annotations, (c) Ideal images with gauge centered.
  • Figure 3: Heatmaps of overall, horizontal, and vertical accuracy showing that most errors occur when the drone is near the image centerline.
  • Figure 4: Sample images where the latent model fails to estimate the best actions to center the object in the image.