Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Seeing, Saying, Solving: An LLM-to-TL Framework for Cooperative Robots

Dan BW Choe, Sundhar Vinodh Sangeetha, Steven Emanuel, Chih-Yuan Chiu, Samuel Coogan, Shreyas Kousik

TL;DR

The paper presents a decentralized framework for cooperative robot conflicts in warehouses, combining vision-language conflict detection with LLM-driven natural language requests and a formal STL/MILP pipeline to ensure safety and feasibility. A BN F grammar constrains NL-to-STL translations, enabling LLM outputs to be directly solved as MILPs for path reconfiguration, while a requester selects the partner that minimizes total system makespan. The approach is validated in NL-to-TL translation experiments showing 100% STL validity and competitive accuracy, and in a forklift-pallet scenario demonstrating meaningful system-time savings over heuristics. This work enables scalable, safe multi-agent coordination without centralized task allocation, with practical impact for time-critical, heterogeneous robot deployments in warehousing.

Abstract

Increased robot deployment, such as in warehousing, has revealed a need for seamless collaboration among heterogeneous robot teams to resolve unforeseen conflicts. To address this challenge, we propose a novel, decentralized framework for robots to request and provide help. The framework begins with robots detecting conflicts using a Vision Language Model (VLM), then reasoning over whether help is needed. If so, it crafts and broadcasts a natural language (NL) help request using a Large Language Model (LLM). Potential helper robots reason over the request and offer help (if able), along with information about impact to their current tasks. Helper reasoning is implemented via an LLM grounded in Signal Temporal Logic (STL) using a Backus-Naur Form (BNF) grammar to guarantee syntactically valid NL-to-STL translations, which are then solved as a Mixed Integer Linear Program (MILP). Finally, the requester robot chooses a helper by reasoning over impact on the overall system. We evaluate our system via experiments considering different strategies for choosing a helper, and find that a requester robot can minimize overall time impact on the system by considering multiple help offers versus simple heuristics (e.g., selecting the nearest robot to help).

Seeing, Saying, Solving: An LLM-to-TL Framework for Cooperative Robots

TL;DR

The paper presents a decentralized framework for cooperative robot conflicts in warehouses, combining vision-language conflict detection with LLM-driven natural language requests and a formal STL/MILP pipeline to ensure safety and feasibility. A BN F grammar constrains NL-to-STL translations, enabling LLM outputs to be directly solved as MILPs for path reconfiguration, while a requester selects the partner that minimizes total system makespan. The approach is validated in NL-to-TL translation experiments showing 100% STL validity and competitive accuracy, and in a forklift-pallet scenario demonstrating meaningful system-time savings over heuristics. This work enables scalable, safe multi-agent coordination without centralized task allocation, with practical impact for time-critical, heterogeneous robot deployments in warehousing.

Abstract

Increased robot deployment, such as in warehousing, has revealed a need for seamless collaboration among heterogeneous robot teams to resolve unforeseen conflicts. To address this challenge, we propose a novel, decentralized framework for robots to request and provide help. The framework begins with robots detecting conflicts using a Vision Language Model (VLM), then reasoning over whether help is needed. If so, it crafts and broadcasts a natural language (NL) help request using a Large Language Model (LLM). Potential helper robots reason over the request and offer help (if able), along with information about impact to their current tasks. Helper reasoning is implemented via an LLM grounded in Signal Temporal Logic (STL) using a Backus-Naur Form (BNF) grammar to guarantee syntactically valid NL-to-STL translations, which are then solved as a Mixed Integer Linear Program (MILP). Finally, the requester robot chooses a helper by reasoning over impact on the overall system. We evaluate our system via experiments considering different strategies for choosing a helper, and find that a requester robot can minimize overall time impact on the system by considering multiple help offers versus simple heuristics (e.g., selecting the nearest robot to help).
Paper Structure (21 sections, 9 equations, 3 figures, 1 table)

This paper contains 21 sections, 9 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries and Problem Statement
  3. Proposed Method
  4. Generating Help Requests, Offers, and Confirmations
  5. Translating Help Proposals to STL Specifications
  6. Solving for Optimal Robot Paths
  7. Experiments
  8. Experiment 1: Natural Language to STL
  9. Experiment 2: Mobile Robot Blocked by a Pallet
  10. Conclusion and Future Work
  11. Encoding STL Specifications
  12. Global STL Specification
  13. Pick-and-Place (PNP) Tasks as STL Specifications
  14. Example LTL BNF Grammar
  15. Natural Language
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Overview of the proposed framework. The requester agent encounters a conflict and broadcasts a help request to other agents if needed. A potential helper agent receives the message and reasons whether it has the required capabilities to assist. If able to assist, the potential helper agent computes the cost of helping and includes it in a help offer sent to the requester. The requester selects the help offer with the lowest cost and sends a confirmation message to the selected helper.
  • Figure 2: An example of a reconfigured path. The help site is reached in 11 time-steps, extending the original path by 4 time-steps for a total cost $\tau^{\mathrm{\textnormal{h}}}_j + \tau^{\mathrm{\textnormal{new}}}_j = 15$ time-steps.
  • Figure 3: Comparison of total time-steps added to the system under heuristics and the optimal solution. The closest helper (H1) coincided with the optimal solution in 42% of trials while the first offer (H2) was optimal in only 25% of trials.