Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AI Assistants for Spaceflight Procedures: Combining Generative Pre-Trained Transformer and Retrieval-Augmented Generation on Knowledge Graphs With Augmented Reality Cues

Oliver Bensch, Leonie Bensch, Tommy Nilsson, Florian Saling, Bernd Bewer, Sophie Jentzsch, Tobias Hecking, J. Nathan Kutz

TL;DR

The paper addresses the challenge of providing robust, offline-capable assistance for astronauts to perform spaceflight procedures during Earth communication outages. It presents CORE, a modular IPA that fuses Generative Pre-Trained Transformers with Retrieval-Augmented Generation over Knowledge Graphs and augments guidance with AR cues to deliver intuitive, 3D spatial procedure support. Key contributions include Graph of Thoughts for transparency, confidence scoring for retrieved data, and support for offline deployment using open-source models such as Llama 3.1, maintaining reliability and user trust. The prototype demonstrates a RAG-enabled procedure retrieval pipeline using NASA ISS procedures, with an offline second version and planned ESA EAC evaluations, highlighting practical viability for Artemis-era missions.

Abstract

This paper describes the capabilities and potential of the intelligent personal assistant (IPA) CORE (Checklist Organizer for Research and Exploration), designed to support astronauts during procedures onboard the International Space Station (ISS), the Lunar Gateway station, and beyond. We reflect on the importance of a reliable and flexible assistant capable of offline operation and highlight the usefulness of audiovisual interaction using augmented reality elements to intuitively display checklist information. We argue that current approaches to the design of IPAs in space operations fall short of meeting these criteria. Therefore, we propose CORE as an assistant that combines Knowledge Graphs (KGs), Retrieval-Augmented Generation (RAG) for a Generative Pre-Trained Transformer (GPT), and Augmented Reality (AR) elements to ensure an intuitive understanding of procedure steps, reliability, offline availability, and flexibility in terms of response style and procedure updates.

AI Assistants for Spaceflight Procedures: Combining Generative Pre-Trained Transformer and Retrieval-Augmented Generation on Knowledge Graphs With Augmented Reality Cues

TL;DR

The paper addresses the challenge of providing robust, offline-capable assistance for astronauts to perform spaceflight procedures during Earth communication outages. It presents CORE, a modular IPA that fuses Generative Pre-Trained Transformers with Retrieval-Augmented Generation over Knowledge Graphs and augments guidance with AR cues to deliver intuitive, 3D spatial procedure support. Key contributions include Graph of Thoughts for transparency, confidence scoring for retrieved data, and support for offline deployment using open-source models such as Llama 3.1, maintaining reliability and user trust. The prototype demonstrates a RAG-enabled procedure retrieval pipeline using NASA ISS procedures, with an offline second version and planned ESA EAC evaluations, highlighting practical viability for Artemis-era missions.

Abstract

This paper describes the capabilities and potential of the intelligent personal assistant (IPA) CORE (Checklist Organizer for Research and Exploration), designed to support astronauts during procedures onboard the International Space Station (ISS), the Lunar Gateway station, and beyond. We reflect on the importance of a reliable and flexible assistant capable of offline operation and highlight the usefulness of audiovisual interaction using augmented reality elements to intuitively display checklist information. We argue that current approaches to the design of IPAs in space operations fall short of meeting these criteria. Therefore, we propose CORE as an assistant that combines Knowledge Graphs (KGs), Retrieval-Augmented Generation (RAG) for a Generative Pre-Trained Transformer (GPT), and Augmented Reality (AR) elements to ensure an intuitive understanding of procedure steps, reliability, offline availability, and flexibility in terms of response style and procedure updates.
Paper Structure (4 sections, 1 figure)

This paper contains 4 sections, 1 figure.

Table of Contents

  1. Introduction
  2. IPAs for Future Space Missions
  3. Results - CORE Prototype
  4. Discussion

Figures (1)

  • Figure 1: Example ISS procedure containing specific steps for a CPR, featuring pictures and location-depended information about the placement of electrodes on the body of the patient NASA2016ISS.