From Abstraction to Reality: DARPA's Vision for Robust Sim-to-Real Autonomy

TL;DR

The paper addresses the persistent sim-to-real transfer gap that plagues autonomous systems operating in dynamic real-world environments. It proposes TIAMAT's abstract-to-real transfer framework, which relies on broad diversity from low-fidelity simulations and semantic anchors to enable rapid, robust adaptation, even as the transfer gap widens. A bidirectional refinement cycle between abstraction and real-world experience, plus a focus on perception, planning, and control, underpin the approach. The program includes two competitions (APSU and Indy) to evaluate abstract-to-real transfer across semantic understanding, dynamics, and missions, with a comprehensive evaluation suite designed to quantify semantic alignment and real-world performance, ultimately aiming for faster deployment and greater resilience in varied platforms and tasks.

Abstract

The DARPA Transfer from Imprecise and Abstract Models to Autonomous Technologies (TIAMAT) program aims to address rapid and robust transfer of autonomy technologies across dynamic and complex environments, goals, and platforms. Existing methods for simulation-to-reality (sim-to-real) transfer often rely on high-fidelity simulations and struggle with broad adaptation, particularly in time-sensitive scenarios. Although many approaches have shown incredible performance at specific tasks, most techniques fall short when posed with unforeseen, complex, and dynamic real-world scenarios due to the inherent limitations of simulation. In contrast to current research that aims to bridge the gap between simulation environments and the real world through increasingly sophisticated simulations and a combination of methods typically assuming a small sim-to-real gap -- such as domain randomization, domain adaptation, imitation learning, meta-learning, policy distillation, and dynamic optimization -- TIAMAT takes a different approach by instead emphasizing transfer and adaptation of the autonomy stack directly to real-world environments by utilizing a breadth of low(er)-fidelity simulations to create broadly effective sim-to-real transfers. By abstractly learning from multiple simulation environments in reference to their shared semantics, TIAMAT's approaches aim to achieve abstract-to-real transfer for effective and rapid real-world adaptation. Furthermore, this program endeavors to improve the overall autonomy pipeline by addressing the inherent challenges in translating simulated behaviors into effective real-world performance.

Figures (4)

• Figure 1: TIAMAT's goal: Increased performance with large transfer gaps
• Figure 2: Abstract to Real Transfer Cycle
• Figure 3: APSU Challenge Timeline
• Figure 4: Performer approaches in the autonomy transfer and refinement loop