AutoM3L: An Automated Multimodal Machine Learning Framework with Large Language Models

TL;DR

AutoM3L presents an LLM-powered framework for automated multimodal machine learning, eliminating manual feature engineering and hyperparameter tuning by orchestrating five modules (Modality Inference, Automated Feature Engineering, Model Selection, Pipeline Assembly, and Hyperparameter Optimization) through tailored LLM prompts. It represents multimodal data as structured tables, enabling cross-modal interaction and using an ensemble of model cards to select modality-appropriate models, followed by a code-generated fusion pipeline and automated HPO. Extensive experiments on six multimodal datasets and open unimodal benchmarks show competitive performance relative to rule-based AutoML methods, with additional evidence from a user study indicating improved usability and reduced learning curve. The work demonstrates that LLM-driven automation can deliver end-to-end multimodal AutoML with interactive customization and potential for broader modality support, while acknowledging biases and cost considerations in LLM usage.

Abstract

Automated Machine Learning (AutoML) offers a promising approach to streamline the training of machine learning models. However, existing AutoML frameworks are often limited to unimodal scenarios and require extensive manual configuration. Recent advancements in Large Language Models (LLMs) have showcased their exceptional abilities in reasoning, interaction, and code generation, presenting an opportunity to develop a more automated and user-friendly framework. To this end, we introduce AutoM3L, an innovative Automated Multimodal Machine Learning framework that leverages LLMs as controllers to automatically construct multimodal training pipelines. AutoM3L comprehends data modalities and selects appropriate models based on user requirements, providing automation and interactivity. By eliminating the need for manual feature engineering and hyperparameter optimization, our framework simplifies user engagement and enables customization through directives, addressing the limitations of previous rule-based AutoML approaches. We evaluate the performance of AutoM3L on six diverse multimodal datasets spanning classification, regression, and retrieval tasks, as well as a comprehensive set of unimodal datasets. The results demonstrate that AutoM3L achieves competitive or superior performance compared to traditional rule-based AutoML methods. Furthermore, a user study highlights the user-friendliness and usability of our framework, compared to the rule-based AutoML methods.

Figures (6)

• Figure 1: The overall framework of AutoM3L. It consists of five stages: ① Infer the modality of each attribute in structured table data. ② Automate feature engineering for feature filtering and data imputation. ③ Select optimal models for each modality. ④ Generates executable scripts for model fusion and data processing to assemble the training pipeline. ⑤ Search optimal hyperparameters. The detailed system prompts for LLMs in each stage can be found in Appendix \ref{['appendix-prompts']}.
• Figure 2: (a) Modality Inference with MI-LLM. It displays MI-LLM's capability to determine the modality of each column in a dataset. Attributes are annotated in red to indicate the inferred modality. (b) Data Refinement with AFE-LLM. It highlights AFE-LLM's dual role in feature filtering and data imputation. The left part displays attributes marked in red that are filtered out, while the right part shows red annotations identifying attributes that undergo imputation.
• Figure 3: Illustration of the model zoo and MS-LLM. (a) Model addition process: This stage showcases how new models are incorporated into the model zoo, visualized as a vector database. The model card's embedding vector serves as the unique identifier or key, paired with its corresponding model configuration as the value. (b) Model retrieval process: This stage illustrates the model selection process. Given user directives, the system initiates a query to identify the top 5 models that align with each input modality. From this refined subset, MS-LLM then determines and selects the most suitable model.
• Figure 4: (a) The PA-LLM is responsible for generating executable code, ensuring seamless model training and data processing. (b) The HPO-LLM deduces hyperparameters and defines search intervals for hyperparameter optimization.
• Figure 5: Boxplots displaying the distribution of the four variables collected in the user study.