LTL-Transfer: Skill Transfer for Temporal Task Specification

TL;DR

This work tackles safe generalization for temporal task specifications in reinforcement learning by addressing zero-shot transfer across novel LTL formulas. It introduces LTL-Transfer, which decomposes training-task policies into transition-centric options that are portable across tasks by leveraging the structure of reward machines and the LTL automaton. The method compiles these options offline and then assembles them online to follow paths in a novel reward machine, employing Constrained or Relaxed edge-matching to preserve safety while expanding applicability. In Minecraft-inspired domains and a real Spot robot, LTL-Transfer achieves high success on unseen tasks (e.g., >90% on 100 challenging unseen tasks after training on 50 tasks) with strict safety guarantees, demonstrating data efficiency and practical impact for safe temporal-task planning.

Abstract

Deploying robots in real-world environments, such as households and manufacturing lines, requires generalization across novel task specifications without violating safety constraints. Linear temporal logic (LTL) is a widely used task specification language with a compositional grammar that naturally induces commonalities among tasks while preserving safety guarantees. However, most prior work on reinforcement learning with LTL specifications treats every new task independently, thus requiring large amounts of training data to generalize. We propose LTL-Transfer, a zero-shot transfer algorithm that composes task-agnostic skills learned during training to safely satisfy a wide variety of novel LTL task specifications. Experiments in Minecraft-inspired domains show that after training on only 50 tasks, LTL-Transfer can solve over 90% of 100 challenging unseen tasks and 100% of 300 commonly used novel tasks without violating any safety constraints. We deployed LTL-Transfer at the task-planning level of a quadruped mobile manipulator to demonstrate its zero-shot transfer ability for fetch-and-deliver and navigation tasks.

Figures (10)

• Figure 1: The robot is executing four task-agnostic skills sequentially to solve a novel task $\mathbf{F} (book \wedge \mathbf{F} (desk_a \wedge \mathbf{F}(juice \wedge \mathbf{F} desk_a)))$, i.e., fetch and deliver a book then a juice bottle to the user. Two of the four skills are shown.
• Figure 1: Figure \ref{['fig:lc_relaxed_hard']} depicts the success rates of transferring to five different specifications types using the Relaxed edge-matching condition after LTL-Transfer being trained on Hard training sets of various sizes. Figure \ref{['fig:lc_constrained_hard']} depicts the success rates with the Constrained edge-matching condition. Note that the error bars depict the 95% credible interval if the successful transfer was modeled as a Bernoulli distribution.
• Figure 2: An example of tasks, the environment, and trajectories. The robot learned to solve the two training LTL tasks and is expected to solve two novel tasks $\varphi_1$ and $\varphi_2$. Figure 2a depicts the trajectories output by a subtask-based algorithm (blue for $\varphi_1$, red for $\varphi_2$). Figure 2b depicts the trajectories produced by our proposed algorithm LTL-Transfer. Note that LTL-Transfer does not start execution for the task $\varphi_2$ as the two learned policies do not guarantee the preservation of the ordering constraint. Figure 2c depicts the optimal trajectories for the novel tasks $\varphi_1$ and $\varphi_2$. Figure 2d is a graph representation of the reward machine (RM) for the task $\varphi_2$. Nodes represent RM states. Edges represent Boolean formulas.
• Figure 2: Figure \ref{['fig:lc_relaxed_soft']} depicts the success rates of transferring to five different specifications types using the Relaxed edge-matching condition after LTL-Transfer being trained on Soft training sets of various sizes. Figure \ref{['fig:lc_constrained_soft']} depicts the success rates with the Constrained edge-matching condition. Note that the error bars depict the 95% credible interval if the successful transfer was modeled as a Bernoulli distribution.
• Figure 3: Figure \ref{['fig:baseline_success']} shows the success rates of four methods on five test sets after training on the Mixed training set. Figure \ref{['fig:baseline_violation']} depicts their specification violation rates (with a shared legend). Figure \ref{['fig:relaxed']} and \ref{['fig:constrained']} show the success rates of LTL-Transfer on five test sets after training on Mixed sets of various sizes using Relaxed and Constrained edge-matching condition, respectively. The error bars depict the 95% credible interval if the successful transfer was modeled as a Bernoulli distribution. Figure \ref{['fig:stochastic']} shows the success rate as the slip probability increases averaged over four maps.