BioCAP: Exploiting Synthetic Captions Beyond Labels in Biological Foundation Models

TL;DR

BioCAP introduces instance-level descriptive captions as complementary supervision for biological multimodal foundation models by grounding synthetic captions in domain knowledge from Wikipedia and taxonomy-guided prompts. The two-view text supervision (taxonomy labels and captions) with separate visual projectors biases image representations toward diagnostic traits, improving zero-shot species classification and text-image retrieval. Through a scalable pipeline generating 132K Wikipedia-grounded descriptions and 896 formatted examples, BioCAP achieves notable performance gains over BioCLIP and CLIP, and ablations show domain contexts and capitalizing on trait-focused prompts are crucial. The work demonstrates that semantically rich, domain-grounded captions bridge biological images with multimodal models, enabling richer biological semantics and practical retrieval capabilities.

Abstract

This work investigates descriptive captions as an additional source of supervision for biological multimodal foundation models. Images and captions can be viewed as complementary samples from the latent morphospace of a species, each capturing certain biological traits. Incorporating captions during training encourages alignment with this shared latent structure, emphasizing potentially diagnostic characters while suppressing spurious correlations. The main challenge, however, lies in obtaining faithful, instance-specific captions at scale. This requirement has limited the utilization of natural language supervision in organismal biology compared with many other scientific domains. We complement this gap by generating synthetic captions with multimodal large language models (MLLMs), guided by Wikipedia-derived visual information and taxon-tailored format examples. These domain-specific contexts help reduce hallucination and yield accurate, instance-based descriptive captions. Using these captions, we train BioCAP (i.e., BioCLIP with Captions), a biological foundation model that captures rich semantics and achieves strong performance in species classification and text-image retrieval. These results demonstrate the value of descriptive captions beyond labels in bridging biological images with multimodal foundation models.

Figures (10)

• Figure 1: Left: Different strategies to create captions for biological images. Wikipedia offers rich domain knowledge, but descriptions are often generic and not directly grounded in the given image. Multimodal large language models (MLLMs) may hallucinate details when conditioned solely by images (wrong color description in this example). Incorporating Wikipedia-derived visual information and taxon-tailored format examples as contexts helps generate accurate, image-specific captions. Right: Using these descriptive captions as additional supervision, BioCAP captures fine-grained biological semantics. Please refer to \ref{['fig:tsne']} for detailed comparisons with other models.
• Figure 2: The pipeline of collecting domain-specific context for MLLMs. We query Wikipedia with the scientific name to get the corresponding webpage. After validating the full taxonomic rank, we process the descriptive excerpts with MLLMs to extract visual information. For each taxonomic class, we randomly select up to three species and curate format examples through Gemini Deep Research. These contexts help MLLMs generate accurate and grounded descriptive captions.
• Figure 3: Captions generated by different strategies (refer to \ref{['tab:abl-caption']} and §\ref{['sec:abl-baseline-prompts']}). The domain-specific contexts of taxon-tailored format examples and Wikipedia-derived visual information significantly reduce hallucination and provide more accurate descriptions of the target object.
• Figure 4: Grad-CAM visualization of CLIP, BioCLIP, and BioCAP, given species names and biological concepts frequently mentioned in their captions. Comparatively, BioCAP offers a comprehensive understanding of these concepts and connects them to the corresponding species.
• Figure 5: Grad-CAM visualization of CLIP, BioCLIP, and BioCAP, given behaviors. BioCAP correctly highlights the body parts related to the behaviors.