Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cytoarchitecture in Words: Weakly Supervised Vision-Language Modeling for Human Brain Microscopy

Matthew Sutton, Katrin Amunts, Timo Dickscheid, Christian Schiffer

TL;DR

A label-mediated method that generates meaningful captions from images by linking images and text only through a label, without requiring curated paired image-text data is proposed, suggesting that weak, label-mediated pairing can suffice to connect existing biomedical vision foundation models to language.

Abstract

Foundation models increasingly offer potential to support interactive, agentic workflows that assist researchers during analysis and interpretation of image data. Such workflows often require coupling vision to language to provide a natural-language interface. However, paired image-text data needed to learn this coupling are scarce and difficult to obtain in many research and clinical settings. One such setting is microscopic analysis of cell-body-stained histological human brain sections, which enables the study of cytoarchitecture: cell density and morphology and their laminar and areal organization. Here, we propose a label-mediated method that generates meaningful captions from images by linking images and text only through a label, without requiring curated paired image-text data. Given the label, we automatically mine area descriptions from related literature and use them as synthetic captions reflecting canonical cytoarchitectonic attributes. An existing cytoarchitectonic vision foundation model (CytoNet) is then coupled to a large language model via an image-to-text training objective, enabling microscopy regions to be described in natural language. Across 57 brain areas, the resulting method produces plausible area-level descriptions and supports open-set use through explicit rejection of unseen areas. It matches the cytoarchitectonic reference label for in-scope patches with 90.6% accuracy and, with the area label masked, its descriptions remain discriminative enough to recover the area in an 8-way test with 68.6% accuracy. These results suggest that weak, label-mediated pairing can suffice to connect existing biomedical vision foundation models to language, providing a practical recipe for integrating natural-language in domains where fine-grained paired annotations are scarce.

Cytoarchitecture in Words: Weakly Supervised Vision-Language Modeling for Human Brain Microscopy

TL;DR

A label-mediated method that generates meaningful captions from images by linking images and text only through a label, without requiring curated paired image-text data is proposed, suggesting that weak, label-mediated pairing can suffice to connect existing biomedical vision foundation models to language.

Abstract

Foundation models increasingly offer potential to support interactive, agentic workflows that assist researchers during analysis and interpretation of image data. Such workflows often require coupling vision to language to provide a natural-language interface. However, paired image-text data needed to learn this coupling are scarce and difficult to obtain in many research and clinical settings. One such setting is microscopic analysis of cell-body-stained histological human brain sections, which enables the study of cytoarchitecture: cell density and morphology and their laminar and areal organization. Here, we propose a label-mediated method that generates meaningful captions from images by linking images and text only through a label, without requiring curated paired image-text data. Given the label, we automatically mine area descriptions from related literature and use them as synthetic captions reflecting canonical cytoarchitectonic attributes. An existing cytoarchitectonic vision foundation model (CytoNet) is then coupled to a large language model via an image-to-text training objective, enabling microscopy regions to be described in natural language. Across 57 brain areas, the resulting method produces plausible area-level descriptions and supports open-set use through explicit rejection of unseen areas. It matches the cytoarchitectonic reference label for in-scope patches with 90.6% accuracy and, with the area label masked, its descriptions remain discriminative enough to recover the area in an 8-way test with 68.6% accuracy. These results suggest that weak, label-mediated pairing can suffice to connect existing biomedical vision foundation models to language, providing a practical recipe for integrating natural-language in domains where fine-grained paired annotations are scarce.
Paper Structure (6 sections, 3 figures, 1 table)

This paper contains 6 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Method
  3. Results
  4. LLM backbone selection via cytoarchitectonic QA
  5. Quality of generated captions
  6. Discussion & conclusion

Figures (3)

  • Figure 1: Architecture overview. Image-conditioned text generation is based on Flamingo alayrac2022flamingo. Condition images are embedded by the frozen vision model, CytoNet schiffer2025cytonet. Embeddings are linearly projected into image tokens. A gated cross-attention block grattafiori2024llama3herdmodels after every 4th encoder block of the Llama-3-8b model grattafiori2024llama3herdmodels conditions text generation.
  • Figure 2: Synthetic caption generation pipeline. Starting from probabilistic cytoarchitectonic maps for each Julich-Brain area Amunts2020, we traverse links in the EBRAINS Knowledge Graph to identify seed publications. We then expand the corpus via citation search (Scopus), filter candidates using area-specific keywords in abstracts, and download full texts via PubMed or ScienceDirect. These are then processed by LLM to extract factual statements about each area, which in turn are processed by LLM into synthetic captions.
  • Figure 3: Examples for generated captions of given microscopic image patches. Captions indicate the respective brain areas, as well as descriptions of area-specific characteristics.