Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AutoGluon-Multimodal (AutoMM): Supercharging Multimodal AutoML with Foundation Models

Zhiqiang Tang, Haoyang Fang, Su Zhou, Taojiannan Yang, Zihan Zhong, Tony Hu, Katrin Kirchhoff, George Karypis

TL;DR

AutoMM addresses the need for a unified AutoML framework capable of leveraging foundation models across multiple modalities. It introduces a late-fusion, three-line fine-tuning approach, a Pandas-based data format, and parameter-efficient fine-tuning to handle large models efficiently. The authors provide a 55-dataset multimodal benchmark and show AutoMM outperforms a unimodal baseline Auto-Keras on basic tasks while remaining competitive on advanced tasks like semantic matching and segmentation, with deployment-friendly tooling including TensorRT support. The work offers a practical, scalable path for practitioners to automate multimodal tasks with minimal custom engineering.

Abstract

AutoGluon-Multimodal (AutoMM) is introduced as an open-source AutoML library designed specifically for multimodal learning. Distinguished by its exceptional ease of use, AutoMM enables fine-tuning of foundation models with just three lines of code. Supporting various modalities including image, text, and tabular data, both independently and in combination, the library offers a comprehensive suite of functionalities spanning classification, regression, object detection, semantic matching, and image segmentation. Experiments across diverse datasets and tasks showcases AutoMM's superior performance in basic classification and regression tasks compared to existing AutoML tools, while also demonstrating competitive results in advanced tasks, aligning with specialized toolboxes designed for such purposes.

AutoGluon-Multimodal (AutoMM): Supercharging Multimodal AutoML with Foundation Models

TL;DR

AutoMM addresses the need for a unified AutoML framework capable of leveraging foundation models across multiple modalities. It introduces a late-fusion, three-line fine-tuning approach, a Pandas-based data format, and parameter-efficient fine-tuning to handle large models efficiently. The authors provide a 55-dataset multimodal benchmark and show AutoMM outperforms a unimodal baseline Auto-Keras on basic tasks while remaining competitive on advanced tasks like semantic matching and segmentation, with deployment-friendly tooling including TensorRT support. The work offers a practical, scalable path for practitioners to automate multimodal tasks with minimal custom engineering.

Abstract

AutoGluon-Multimodal (AutoMM) is introduced as an open-source AutoML library designed specifically for multimodal learning. Distinguished by its exceptional ease of use, AutoMM enables fine-tuning of foundation models with just three lines of code. Supporting various modalities including image, text, and tabular data, both independently and in combination, the library offers a comprehensive suite of functionalities spanning classification, regression, object detection, semantic matching, and image segmentation. Experiments across diverse datasets and tasks showcases AutoMM's superior performance in basic classification and regression tasks compared to existing AutoML tools, while also demonstrating competitive results in advanced tasks, aligning with specialized toolboxes designed for such purposes.
Paper Structure (16 sections, 5 figures, 3 tables)

This paper contains 16 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. AutoMM
  4. Data Format and Processing
  5. APIs
  6. Models
  7. Training
  8. Deployment
  9. Advanced Tasks
  10. Experiments
  11. Classification and Regression
  12. Semantic Matching
  13. Object Detection
  14. Semantic Segmentation
  15. Conclusion and Future Work
  16. ...and 1 more sections

Figures (5)

  • Figure 1: AutoMM introduction: Supporting both unimodal and multimodal data (Left), AutoMM enables seamless fine-tuning of foundation models (Middle) for basic classification/regression as well as advanced tasks (Right).
  • Figure 2: Interdependency among data modules: Each module from left to right serves as a prerequisite for the subsequent one.
  • Figure 3: Late-fusion model.
  • Figure 4: Bi-encoder model.
  • Figure 5: Comparison of AutoMM and Sentence-Transformer on Text to Text Matching (TTM), Image to Image Matching (IIM), Text to Image Matching (TIM), and Image to Text Matching (ITM).