Improving Large Vision-Language Models' Understanding for Flow Field Data

TL;DR

This work introduces FieldLVLM, a novel framework designed to improve large vision-language models' understanding of field data, and suggests that this approach opens up new possibilities for applying large vision-language models to scientific research.

Abstract

Large Vision-Language Models (LVLMs) have shown impressive capabilities across a range of tasks that integrate visual and textual understanding, such as image captioning and visual question answering. These models are trained on large-scale image and video datasets paired with text, enabling them to bridge visual perception and natural language processing. However, their application to scientific domains, especially in interpreting complex field data commonly used in the natural sciences, remains underexplored. In this work, we introduce FieldLVLM, a novel framework designed to improve large vision-language models' understanding of field data. FieldLVLM consists of two main components: a field-aware language generation strategy and a data-compressed multimodal model tuning. The field-aware language generation strategy leverages a special-purpose machine learning pipeline to extract key physical features from field data, such as flow classification, Reynolds number, and vortex patterns. This information is then converted into structured textual descriptions that serve as a dataset. The data-compressed multimodal model tuning focuses on LVLMs with these generated datasets, using a data compression strategy to reduce the complexity of field inputs and retain only the most informative values. This ensures compatibility with the models language decoder and guides its learning more effectively. Experimental results on newly proposed benchmark datasets demonstrate that FieldLVLM significantly outperforms existing methods in tasks involving scientific field data. Our findings suggest that this approach opens up new possibilities for applying large vision-language models to scientific research, helping bridge the gap between large models and domain-specific discovery.

Figures (6)

• Figure 1: (a) Quantitative comparison of vision-language responses in field data across different methods. (b) The performance of key evaluation metrics, including flow categorization, Reynolds number calculation, vortex identification, and comprehensive field data interpretation.
• Figure 2: The pipeline of field-aware language data generation strategy integrating special-purpose machine learning models for field classification, Reynolds number estimation and vortex detection.
• Figure 3: Illustrative examples of generated field language representations, showing structured text outputs for flow field analysis derived from velocity and pressure data.
• Figure 4: Input-output architecture of the data-compressed multimodal model featuring VQGAN-based token compression, key value selection, and image representation conversion for enhanced field data semantic analysis.
• Figure 5: Q&A analysis on vortex shedding dynamics and pressure distribution in flow past a bluff body highlighting kármán vortex street characteristics and three-stage flow structure.