Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HIFICL: High-Fidelity In-Context Learning for Multimodal Tasks

Xiaoyu Li, Yuhang Liu, Zheng Luo, Xuanshuo Kang, Fangqi Lou, Xiaohua Wu, Zihan Xiong

Abstract

In-Context Learning (ICL) is a significant paradigm for Large Multimodal Models (LMMs), using a few in-context demonstrations (ICDs) for new task adaptation. However, its performance is sensitive to demonstration configurations and computationally expensive. Mathematically, the influence of these demonstrations can be decomposed into a dynamic mixture of the standard attention output and the context values. Current approximation methods simplify this process by learning a "shift vector". Inspired by the exact decomposition, we introduce High-Fidelity In-Context Learning (HIFICL) to more faithfully model the ICL mechanism. HIFICL consists of three key components: 1) a set of "virtual key-value pairs" to act as a learnable context, 2) a low-rank factorization for stable and regularized training, and 3) a simple end-to-end training objective. From another perspective, this mechanism constitutes a form of context-aware Parameter-Efficient Fine-Tuning (PEFT). Extensive experiments show that HiFICL consistently outperforms existing approximation methods on several multimodal benchmarks. The code is available at https://github.com/bbbandari/HiFICL.

HIFICL: High-Fidelity In-Context Learning for Multimodal Tasks

Abstract

In-Context Learning (ICL) is a significant paradigm for Large Multimodal Models (LMMs), using a few in-context demonstrations (ICDs) for new task adaptation. However, its performance is sensitive to demonstration configurations and computationally expensive. Mathematically, the influence of these demonstrations can be decomposed into a dynamic mixture of the standard attention output and the context values. Current approximation methods simplify this process by learning a "shift vector". Inspired by the exact decomposition, we introduce High-Fidelity In-Context Learning (HIFICL) to more faithfully model the ICL mechanism. HIFICL consists of three key components: 1) a set of "virtual key-value pairs" to act as a learnable context, 2) a low-rank factorization for stable and regularized training, and 3) a simple end-to-end training objective. From another perspective, this mechanism constitutes a form of context-aware Parameter-Efficient Fine-Tuning (PEFT). Extensive experiments show that HiFICL consistently outperforms existing approximation methods on several multimodal benchmarks. The code is available at https://github.com/bbbandari/HiFICL.
Paper Structure (35 sections, 4 equations, 10 figures, 7 tables)

This paper contains 35 sections, 4 equations, 10 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. ICL in Large Multimodal Models
  4. The Linear Shift Approximation
  5. Low-Rank Adaptation for PEFT
  6. Methodology
  7. Mathematical Analysis
  8. HiFICL: Direct Parameterization
  9. Architectural Comparison
  10. HiFICL as a Dynamic PEFT
  11. Experiments
  12. Setup
  13. Models, Datasets, and Metrics.
  14. Implementation Details.
  15. Comparison with Existing Methods
  16. ...and 20 more sections

Figures (10)

  • Figure 1: Overview of the HiFICL framework. (a) The overall training process involves a frozen LMM backbone and layer-wise trainable virtual key-value pairs, optimized via a final task loss. (b) Inside each attention head, these virtual pairs are used to compute the high-fidelity attention output according to our derived formula (\ref{['eq:decomposition']}). (c) Conceptually, our dynamic, query-dependent approximation (yellow line) provides a more faithful fit to the underlying data manifold compared to previous linear shift approximations (blue line).
  • Figure 2: Architectural comparison of HiFICL and MimIC. (a) HiFICL implements the full, non-linear attention dynamic. (b) MimIC simplifies the ICL effect into a uni-directional linear shift.
  • Figure 3: Illustration of the Low-Rank Adaptation (LoRA) mechanism. LoRA injects static, trainable low-rank updates ($\Delta W$) into the frozen weight matrices of an attention module.
  • Figure 4: Data efficiency on VQAv2 and COCO across different models and training set sizes.
  • Figure 5: Training efficiency comparison on the Idefics2 model. Bars show the relative cost of MimIC normalized against our method HiFICL (Baseline=1.0x). MimIC's paradigm incurs substantial overhead.
  • ...and 5 more figures