Batch Augmentation with Unimodal Fine-tuning for Multimodal Learning

TL;DR

This work tackles fetal organ detection from ultrasound by leveraging multimodal data (images and accompanying textual/numeric descriptions) through batch augmentation and unimodal fine-tuning. It introduces a transfer-learning pipeline that pre-trains initial layers on medical data, then applies per-sample random augmentations during multimodal training, and fuses image features with text/numeric descriptors in a dedicated head. Evaluations on the FPU23 and UPMC Food-101 datasets show strong performance, including near-SOTA results on Food-101 (92.63% test accuracy) and high detection accuracies across head, abdomen, arm, and leg tasks on FPU23 with ViT-L/16 and the proposed training regime. The approach promises practical clinical impact by enabling robust, real-time multimodal fetal assessment and offers avenues for extending to fetal age prediction and broader medical datasets, with open-source scripts provided.

Abstract

This paper proposes batch augmentation with unimodal fine-tuning to detect the fetus's organs from ultrasound images and associated clinical textual information. We also prescribe pre-training initial layers with investigated medical data before the multimodal training. At first, we apply a transferred initialization with the unimodal image portion of the dataset with batch augmentation. This step adjusts the initial layer weights for medical data. Then, we apply neural networks (NNs) with fine-tuned initial layers to images in batches with batch augmentation to obtain features. We also extract information from descriptions of images. We combine this information with features obtained from images to train the head layer. We write a dataloader script to load the multimodal data and use existing unimodal image augmentation techniques with batch augmentation for the multimodal data. The dataloader brings a new random augmentation for each batch to get a good generalization. We investigate the FPU23 ultrasound and UPMC Food-101 multimodal datasets. The multimodal large language model (LLM) with the proposed training provides the best results among the investigated methods. We receive near state-of-the-art (SOTA) performance on the UPMC Food-101 dataset. We share the scripts of the proposed method with traditional counterparts at the following repository: github.com/dipuk0506/multimodal

Figures (7)

• Figure 1: Information flow in the proposed multimodal learning. We extract features from images through pre-trained initial layers. We also extract information from descriptions of images. Moreover, we extract labels from descriptions. This figure shows a vision transformer (ViT) used to obtain image features. We also apply a ResNet-type model for the feature extraction.
• Figure 2: Visualization of the importance of batch augmentation. Rough diagram (a) presents samples of different classes with different shapes. The green dotted lines represent the ground truth decision boundary. Red solid lines show the decision boundary of a poorly trained NN. Rough diagram (b) presents the effect while all samples in a batch receive the same augmentation. Rough diagram (c) presents the situation, while different samples receive different augmentations. Patterns with thick solid lines present original samples, and patterns with thin dotted lines present augmented samples.
• Figure 3: Two example image on the FPU23 dataset. (a) The image contains the abdomen and two arms. (b) The image contains the head and the abdomen.
• Figure 4: An example sample on the UPMC Food-101 dataset. Each sample contains an image, a title, and a label.
• Figure 5: Visualization of all images in 32 images after augmentation on the FPU23 dataset. We apply random rotations, random crops, random horizontal flips, and random vertical flips augmentations on training images.