Space-LLaVA: a Vision-Language Model Adapted to Extraterrestrial Applications

TL;DR

Space-LLaVA introduces a first-step space foundation model by adapting a strongVision-Language Model (LLaVA-13B) to extraterrestrial data through synthetically annotated, multi-dataset QA and instruction tasks. The approach leverages GPT-assisted augmentation of AI4Mars, MICD, and SpaceScienceQA to create a rich training corpus, and demonstrates that joint fine-tuning of the language backbone and vision-language adapter yields the best performance, including emergent abilities on unseen tasks while mitigating catastrophic forgetting with a portion of pre-training data. Evaluations span in-distribution, out-of-distribution (orbit), and embodied lunar-simulation settings, showing notable gains over zero-shot baselines and highlighting both the potential and the remaining challenges of deploying FMs in space robotics. The work also explores integration into modular autonomy stacks, illustrating how space-oriented FMs can function as high-level planners and runtime monitors, thereby enabling more autonomous and scalable space missions. Overall, Space-LLaVA provides a practical blueprint for adapting generalist foundation models to domain-specific, multimodal space tasks and sets a path for expanding space-domain datasets and encoders to support robust, autonomous space operations.

Abstract

Foundation Models (FMs), e.g., large language models, possess attributes of intelligence which offer promise to endow a robot with the contextual understanding necessary to navigate complex, unstructured tasks in the wild. We see three core challenges in the future of space robotics that motivate building an FM for the space robotics community: 1) Scalability of ground-in-the-loop operations; 2) Generalizing prior knowledge to novel environments; and 3) Multi-modality in tasks and sensor data. As a first-step towards a space foundation model, we programmatically augment three extraterrestrial databases with fine-grained language annotations inspired by the sensory reasoning necessary to e.g., identify a site of scientific interest on Mars, building a synthetic dataset of visual-question-answer and visual instruction-following tuples. We fine-tune a pre-trained LLaVA 13B checkpoint on our augmented dataset to adapt a Vision-Language Model (VLM) to the visual semantic features in an extraterrestrial environment, demonstrating FMs as a tool for specialization and enhancing a VLM's zero-shot performance on unseen task types in comparison to state-of-the-art VLMs. Ablation studies show that fine-tuning the language backbone and vision-language adapter in concert is key to facilitate adaption while a small percentage, e.g., 20%, of the pre-training data can be used to safeguard against catastrophic forgetting.

Figures (18)

• Figure 1: Space-LLaVA outperforms SoTA VLMs, e.g., GPT-4o openai_gpt4o and base LLaVA Liu2023ImprovedBW, annotating withheld observations from our synthetic dataset of extraterrestrial, planetary imagery and learns to service queries on an unseen task type as an emergent ability.
• Figure 2: We present Space-LLaVA, initialized from a pre-trained LLaVA 13B model Liu2023ImprovedBW and fine-tuned to extraterrestrial applications with our synthetically generated dataset of, e.g., instruction-following, conversations constructed from three extraterrestrial datasets. This model accepts two data modalities: RGB images and text. Each image is mapped into a shared latent space by the model's image encoder and multi-modal adapter from which a large language model produces a response in natural language. As such, our general-purpose model can be used, among other tasks, as a tool for language annotation servicing requests previously withheld from training.
• Figure 3: The AI4Mars dataset ono2021ai4mars provides access to image captures of Mars' terrain with crowd-sourced annotations for four terrain classes: "regolith", "sand", "bedrock", "large rock(s)". Terrain beyond 30m is left unlabeled.
• Figure 4: Sample from the Martian Image Caption Dataset (MICD): "conglomerate outcrops and float rocks and regolith."
• Figure 5: The proportional representation of prompt style, e.g., instruction-following, and the designed fine-tuning tasks, e.g., grain characterization, in our Space-LLaVA dataset. All instruction and VQA-based tasks are derived from the AI4Mars & MICD datasets, while the SpaceScienceQA dataset represents the only language QA-based category.