Multimodal Large Language Models for Image, Text, and Speech Data Augmentation: A Survey

TL;DR

This survey addresses the challenge of enhancing dataset diversity across image, text, and speech through multimodal LLM based data augmentation. It synthesizes recent literature into process-oriented pipelines for image, text, and speech augmentation, cataloging methods, limitations, and proposed remedies across modalities. It provides cross-modal comparisons and identifies gaps such as alignment, data quality, and computational demands, offering a roadmap including reinforcement learning driven approaches like DeepSeek R1 and multimodal pretraining. The findings aim to guide researchers and practitioners in building robust, scalable data augmentation pipelines for real-world deep learning applications.

Abstract

In the past five years, research has shifted from traditional Machine Learning (ML) and Deep Learning (DL) approaches to leveraging Large Language Models (LLMs) , including multimodality, for data augmentation to enhance generalization, and combat overfitting in training deep convolutional neural networks. However, while existing surveys predominantly focus on ML and DL techniques or limited modalities (text or images), a gap remains in addressing the latest advancements and multi-modal applications of LLM-based methods. This survey fills that gap by exploring recent literature utilizing multimodal LLMs to augment image, text, and audio data, offering a comprehensive understanding of these processes. We outlined various methods employed in the LLM-based image, text and speech augmentation, and discussed the limitations identified in current approaches. Additionally, we identified potential solutions to these limitations from the literature to enhance the efficacy of data augmentation practices using multimodal LLMs. This survey serves as a foundation for future research, aiming to refine and expand the use of multimodal LLMs in enhancing dataset quality and diversity for deep learning applications. (Surveyed Paper GitHub Repo: https://github.com/WSUAgRobotics/data-aug-multi-modal-llm. Keywords: LLM data augmentation, Grok text data augmentation, DeepSeek image data augmentation, Grok speech data augmentation, GPT audio augmentation, voice augmentation, DeepSeek for data augmentation, DeepSeek R1 text data augmentation, DeepSeek R1 image augmentation, Image Augmentation using LLM, Text Augmentation using LLM, LLM data augmentation for deep learning applications)

Figures (8)

• Figure 1: Evolution of data augmentation techniques (from top to bottom) a) Manual transformation functions like image rotation for training dataset expansion. b) LSTM-based automation generating synthetic data. c) Use of Generative LLMs for advanced, context-aware synthetic data creation, marking a shift to AI-driven data augmentation methods.
• Figure 2: Survey methodology and results overview: (a) Flowchart of the structured survey method, showing specific search keywords and steps from initial search to study selection; (b) Graph displaying initial search outcomes across the DBSL database, illustrating paper distribution; (c) Pie chart of selected peer-reviewed papers, showing thorough screening; (d) Pie chart of preprints distribution, indicating extensive preliminary research review. Keywords for image, text, and speech augmentation are marked in red, green, and blue, respectively.
• Figure 3: A comprehensive overview of data augmentation techniques, divided into two main eras: 1990 to 2010, focusing on traditional methods for image, text, and audio augmentation, and 2010 to 2020, highlighting m achine learning and deep learning-based advancements.
• Figure 4: LLM based image data augmentation : showing the technical aspect of how image augmentation using LLM is performed, the techniques of augmenting images using LLM and their limitations
• Figure 5: Conceptual example of LLM-based 3D augmentation. Given a base object, a text prompt could specify modifications like scaling. Adopted from ganeshan2024parselparameterizedshapeediting