Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models

TL;DR

The work tackles robustness and cross-modal alignment in large multimodal models by introducing shuffle-based pretraining and fine-tuning tasks for image and text order reconstruction, augmented with a directed-token mechanism and an attention-guided loss to enhance visual grounding. The Direct-LLaVA framework extends LLaVA by using a directed token to synthesize multimodal cues and applying an Image-to-Response Guided Learning objective to strengthen visual-to-text attention. Empirical results show state-of-the-art performance on both academic task benchmarks and instruction-following LMM benchmarks across multiple backbones, with ablations confirming the benefits of directed tokens, end-sequence placement, permutation diversity, and the grounding loss. This approach advances robust multimodal reasoning and alignment, with practical implications for reliable visual reasoning in instruction-tuned systems; future work includes balancing objective weights and scaling to larger models and data.

Abstract

Large multimodal models (LMMs) have gained impressive performance due to their outstanding capability in various understanding tasks. However, these models still suffer from some fundamental limitations related to robustness and generalization due to the alignment and correlation between visual and textual features. In this paper, we introduce a simple but efficient learning mechanism for improving the robust alignment between visual and textual modalities by solving shuffling problems. In particular, the proposed approach can improve reasoning capability, visual understanding, and cross-modality alignment by introducing two new tasks: reconstructing the image order and the text order into the LMM's pre-training and fine-tuning phases. In addition, we propose a new directed-token approach to capture visual and textual knowledge, enabling the capability to reconstruct the correct order of visual inputs. Then, we introduce a new Image-to-Response Guided loss to further improve the visual understanding of the LMM in its responses. The proposed approach consistently achieves state-of-the-art (SoTA) performance compared with prior LMMs on academic task-oriented and instruction-following LMM benchmarks.

Figures (7)

• Figure 1: Our Proposed Direct-LLaVA achieves State-of-the-Art performance on various LMM benchmarks (Top) with new shuffle learning approaches of Image (Bottom Left) and Text (Bottom Right) during pre-training and fine-tuning phases.
• Figure 2: The Example of Answer Produced by LLaVA v1.5 liu2024improved. The first two rows are samples selected from ScienceQA-IMG, and the last two rows are sampled from MMMU.
• Figure 3: The Proposed Direct-LLaVA Framework. (A) Reconstructing Image Order Task. (B) Reconstructing Text Order Task. (C) Directed-Token Modeling Approach in Image Order Reconstruction. (D) Autoregressive Modeling Approach in Text Order Reconstruction.
• Figure 4: (A) Traditional Single-Turn Conversation Task. (B) Reconstructing Image Ordering Task. (C) Reconstructing Text Ordering Task.
• Figure 5: Comparison of Response in Multiple-Choice Questions Between Direct-LLaVA-7B and LLaVA-v1.5-7B on MMMU-Val.