Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distilling Implicit Multimodal Knowledge into Large Language Models for Zero-Resource Dialogue Generation

Bo Zhang, Hui Ma, Jian Ding, Jian Wang, Bo Xu, Hongfei Lin

TL;DR

This work tackles zero-resource dialogue generation by leveraging implicit visual knowledge distilled from large image-text corpora. It introduces VIKDF, a two-stage framework that first distills visual cues with an Implicit Query Transformer (IQ-Former) and then fuses them into a frozen LLM via Bidirectional Variational Information Fusion (BVIF). The approach demonstrates state-of-the-art performance on zero-resource benchmarks (Image-Chat and Reddit Conversation), supported by comprehensive automatic and human evaluations, ablations, and case studies. The results highlight the value of implicit multimodal signals for enriching dialogue generation when explicit multimodal data is unavailable, with evidence of robustness across different LLM backbones and strong generalization. Overall, VIKDF advances multimodal dialogue research by showing how implicit knowledge can be effectively learned and integrated to produce coherent, relevant, and informative conversations in resource-constrained settings.

Abstract

Integrating multimodal knowledge into large language models (LLMs) represents a significant advancement in dialogue generation capabilities. However, the effective incorporation of such knowledge in zero-resource scenarios remains a substantial challenge due to the scarcity of diverse, high-quality dialogue datasets. To address this, we propose the Visual Implicit Knowledge Distillation Framework (VIKDF), an innovative approach aimed at enhancing LLMs for enriched dialogue generation in zero-resource contexts by leveraging implicit multimodal knowledge. VIKDF comprises two main stages: knowledge distillation, using an Implicit Query Transformer to extract and encode visual implicit knowledge from image-text pairs into knowledge vectors; and knowledge integration, employing a novel Bidirectional Variational Information Fusion technique to seamlessly integrate these distilled vectors into LLMs. This enables the LLMs to generate dialogues that are not only coherent and engaging but also exhibit a deep understanding of the context through implicit multimodal cues, effectively overcoming the limitations of zero-resource scenarios. Our extensive experimentation across two dialogue datasets shows that VIKDF outperforms existing state-of-the-art models in generating high-quality dialogues. The code is available at https://github.com/zhangbo-nlp/VIKDF.

Distilling Implicit Multimodal Knowledge into Large Language Models for Zero-Resource Dialogue Generation

TL;DR

This work tackles zero-resource dialogue generation by leveraging implicit visual knowledge distilled from large image-text corpora. It introduces VIKDF, a two-stage framework that first distills visual cues with an Implicit Query Transformer (IQ-Former) and then fuses them into a frozen LLM via Bidirectional Variational Information Fusion (BVIF). The approach demonstrates state-of-the-art performance on zero-resource benchmarks (Image-Chat and Reddit Conversation), supported by comprehensive automatic and human evaluations, ablations, and case studies. The results highlight the value of implicit multimodal signals for enriching dialogue generation when explicit multimodal data is unavailable, with evidence of robustness across different LLM backbones and strong generalization. Overall, VIKDF advances multimodal dialogue research by showing how implicit knowledge can be effectively learned and integrated to produce coherent, relevant, and informative conversations in resource-constrained settings.

Abstract

Integrating multimodal knowledge into large language models (LLMs) represents a significant advancement in dialogue generation capabilities. However, the effective incorporation of such knowledge in zero-resource scenarios remains a substantial challenge due to the scarcity of diverse, high-quality dialogue datasets. To address this, we propose the Visual Implicit Knowledge Distillation Framework (VIKDF), an innovative approach aimed at enhancing LLMs for enriched dialogue generation in zero-resource contexts by leveraging implicit multimodal knowledge. VIKDF comprises two main stages: knowledge distillation, using an Implicit Query Transformer to extract and encode visual implicit knowledge from image-text pairs into knowledge vectors; and knowledge integration, employing a novel Bidirectional Variational Information Fusion technique to seamlessly integrate these distilled vectors into LLMs. This enables the LLMs to generate dialogues that are not only coherent and engaging but also exhibit a deep understanding of the context through implicit multimodal cues, effectively overcoming the limitations of zero-resource scenarios. Our extensive experimentation across two dialogue datasets shows that VIKDF outperforms existing state-of-the-art models in generating high-quality dialogues. The code is available at https://github.com/zhangbo-nlp/VIKDF.
Paper Structure (31 sections, 9 equations, 7 figures, 4 tables)

This paper contains 31 sections, 9 equations, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Dialogue Generation
  4. Multimodal Knowledge Distillation and Integration
  5. Methodology
  6. Task Formalization and Model Architecture
  7. Text Encoder and Image Encoder
  8. Implicit Query Transformer
  9. Large Language Model
  10. Knowledge Distillation Stage
  11. Text-Image Matching
  12. Text-Assisted Masked Image Modeling
  13. Image-Assisted Masked Text Modeling
  14. Knowledge Integration Stage
  15. Instruction-aware Contextual Inference
  16. ...and 16 more sections

Figures (7)

  • Figure 1: Intuition of our proposed approach.
  • Figure 2: The architecture of IQ-Former.
  • Figure 3: Overview of the knowledge distillation stage. The central part illustrates Text-Image Matching. Below the dashed line lies Text-Assisted Masked Image Modeling, and above, Image-Assisted Masked Text Modeling.
  • Figure 4: Overview of the knowledge integration stage. Instruction-aware Contextual Inference on the left and Instruction-aware Knowledge Reconstruction on the right
  • Figure 5: Case study on Image-Chat test set in a fully zero-resource scenario.
  • ...and 2 more figures