Autonomous Reasoning for Spacecraft Control: A Large Language Model Framework with Group Relative Policy Optimization

TL;DR

The paper tackles autonomous spacecraft control by integrating reasoning-enabled Large Language Models (LLMs) with Group Relative Policy Optimization (GRPO). It introduces a two-stage training pipeline—Supervised Fine-Tuning (SFT) to learn control formatting and primitives, then GRPO to refine policies through interaction—validated across four progressively complex dynamical systems, from linear to nonlinear 3D attitude control. The approach yields feasible, stabilizing policies while providing human-readable reasoning traces, and demonstrates sample-efficient learning via GRPO without a separate value network. Results indicate robust performance and interpretability across linear, nonlinear, orbital, and gyroscopically coupled dynamics, highlighting potential for aerospace autonomy and safety-critical applications. The framework paves the way for hardware-validated, reasoning-driven autonomous control and integration with traditional control approaches like model predictive control.

Abstract

This paper presents a learning-based guidance-and-control approach that couples a reasoning-enabled Large Language Model (LLM) with Group Relative Policy Optimization (GRPO). A two-stage procedure consisting of Supervised Fine-Tuning (SFT) to learn formatting and control primitives, followed by GRPO for interaction-driven policy improvement, trains controllers for each environment. The framework is demonstrated on four control problems spanning a gradient of dynamical complexity, from canonical linear systems through nonlinear oscillatory dynamics to three-dimensional spacecraft attitude control with gyroscopic coupling and thrust constraints. Results demonstrate that an LLM with explicit reasoning, optimized via GRPO, can synthesize feasible stabilizing policies under consistent training settings across both linear and nonlinear systems. The two-stage training methodology enables models to generate control sequences while providing human-readable explanations of their decision-making process. This work establishes a foundation for applying GRPO-based reasoning to autonomous control systems, with potential applications in aerospace and other safety-critical domains.

Figures (7)

• Figure 1: Enhanced system architecture for LLM-based control with two-stage training. The simulation environment provides dynamic system context and feedback. The core LLM system handles runtime inference through state encoding, LLM processing, and action decoding. The two-stage training system implements sequential supervised fine-tuning (SFT) for format learning, followed by Group Relative Policy Optimization (GRPO) for policy refinement.
• Figure 2: Double integrator control performance: (a) GRPO training progression showing reward convergence, (b) control input trajectories, (c) position state evolution, and (d) velocity state evolution.
• Figure 3: Van der Pol oscillator control performance: (a) GRPO training progression with nonlinear dynamics, (b) control input trajectories for limit cycle suppression, (c) position state evolution, and (d) velocity state evolution.
• Figure 4: Orbit raising GRPO training progression showing reward convergence for orbital transfer control.
• Figure 5: Orbital transfer trajectories: (a) radial distance evolution, (b) radial velocity component, (c) tangential velocity component, and (d) thrust angle control inputs.