Can LLMs Do Rocket Science? Exploring the Limits of Complex Reasoning with GTOC 12

TL;DR

This study benchmarks contemporary LLMs on GTOC 12 to probe their capability for autonomous, multi-stage aerospace planning under strict physical constraints. By adapting MLE-Bench and employing the AIDE framework with an LLM-as-a-judge, the work reveals a substantial rise in strategic viability over time, but also an enduring implementation barrier driven by unit inconsistencies, debugging loops, and formatting sensitivity. The findings indicate that current LLMs function as powerful domain facilitators rather than fully autonomous engineers, suggesting that future progress will rely on improved self-correction, expert alignment, and tool-assisted optimization. The work thus offers a pragmatic view of the current limits and points toward concrete directions for integrating reasoning, observability, and domain-specific tooling to enable autonomous mission design. The results have implications for how AI agents may assist high-stakes engineering tasks where correctness and traceability are critical.

Abstract

Large Language Models (LLMs) have demonstrated remarkable proficiency in code generation and general reasoning, yet their capacity for autonomous multi-stage planning in high-dimensional, physically constrained environments remains an open research question. This study investigates the limits of current AI agents by evaluating them against the 12th Global Trajectory Optimization Competition (GTOC 12), a complex astrodynamics challenge requiring the design of a large-scale asteroid mining campaign. We adapt the MLE-Bench framework to the domain of orbital mechanics and deploy an AIDE-based agent architecture to autonomously generate and refine mission solutions. To assess performance beyond binary validity, we employ an "LLM-as-a-Judge" methodology, utilizing a rubric developed by domain experts to evaluate strategic viability across five structural categories. A comparative analysis of models, ranging from GPT-4-Turbo to reasoning-enhanced architectures like Gemini 2.5 Pro, and o3, reveals a significant trend: the average strategic viability score has nearly doubled in the last two years (rising from 9.3 to 17.2 out of 26). However, we identify a critical capability gap between strategy and execution. While advanced models demonstrate sophisticated conceptual understanding, correctly framing objective functions and mission architectures, they consistently fail at implementation due to physical unit inconsistencies, boundary condition errors, and inefficient debugging loops. We conclude that, while current LLMs often demonstrate sufficient knowledge and intelligence to tackle space science tasks, they remain limited by an implementation barrier, functioning as powerful domain facilitators rather than fully autonomous engineers.

Figures (3)

• Figure 1: Schematic of asteroid mining by multiple Mining Shipszhang2025sustainable
• Figure 2: Experiments Results
• Figure 3: Performance evolution of LLMs in the GTOC 12 Strategy Evaluation.