FathomGPT: A Natural Language Interface for Interactively Exploring Ocean Science Data

TL;DR

FathomGPT delivers a natural language interface that enables free-form exploration of ocean science data in FathomNet by integrating a prompt evaluator, knowledge-graph–based name resolution, specialized text-to-SQL models, and Plotly visualizations. The system emphasizes low latency, robust error handling, and session memory to support interactive data exploration, including image-pattern searches and pattern-based retrieval. Through ablation studies and KK-name resolution benchmarks, the authors demonstrate improvements from fine-tuning and context-aware prompting, and show the KG-based approach outperforms alternatives like GPT-4o and vector embeddings for name resolution. The work highlights practical usage scenarios and workshop feedback, arguing that FathomGPT can accelerate marine data analysis and potentially generalize to other scientific databases and domains.

Abstract

We introduce FathomGPT, an open source system for the interactive investigation of ocean science data via a natural language interface. FathomGPT was developed in close collaboration with marine scientists to enable researchers to explore and analyze the FathomNet image database. FathomGPT provides a custom information retrieval pipeline that leverages OpenAI's large language models to enable: the creation of complex queries to retrieve images, taxonomic information, and scientific measurements; mapping common names and morphological features to scientific names; generating interactive charts on demand; and searching by image or specified patterns within an image. In designing FathomGPT, particular emphasis was placed on enhancing the user's experience by facilitating free-form exploration and optimizing response times. We present an architectural overview and implementation details of FathomGPT, along with a series of ablation studies that demonstrate the effectiveness of our approach to name resolution, fine tuning, and prompt modification. We also present usage scenarios of interactive data exploration sessions and document feedback from ocean scientists and machine learning experts.

Figures (14)

• Figure 1: This figure shows four successive screenshots from a single session using the FathomGPT natural language interface for exploring ocean science data. (1) The user asks FathomGPT to return the scientific names for predators of the species commonly known as "moon jelly," and then asks to see images of Mola mola, one of the predators. (2) The user then clicks on an image to see more details about the taxonomy of the species, as well as to learn where and when that particular image was taken and what scientific measurements were gathered at that time. (3) The user then asks FathomGPT to generate an interactive heatmap showing other observations of Mola mola in the Monterey Bay region. (4) Finally, the user asks FathomGPT to generate a detailed scatterplot comparing the temperature and pressure levels at the locations where Mola mola ("ocean sunfish") and another predator Phacellophara camtschatica ("egg-yolk jellyfish") have been observed. FathomGPT facilitates free-form exploration of ocean science data from multiple starting points to support a wide range of research inquiries.
• Figure 2: The FathomGPT pipeline. Here we show a high-level schematic of how an input prompt is processed by the system to produce a JSON output response, which is sent to the frontend webpage to be rendered.
• Figure 3: Example of a Species KG. Here we show the KG containing various characteristics for the species Aurelia aurita, including its common names, morphological features, colors, habitats, diet, and predators.
• Figure 4: An example of KG subject matching. In the input prompt KG, the subject “moon jelly” matches the alias of Aurelia aurita in the species KG. The relation “predators” matches the same relation in the species KG. The result is the list of terms indexed by Data: [“Aurelia aurita”][“predators”].
• Figure 5: An example of KG object matching. In the input prompt KG, the relation “color” matches the same relation in the species KG and the object “transparent” matches one of the colors of Aurelia aurita in the species KG. The result would be the parent concept “Aurelia aurita”.