Bridging Modalities and Transferring Knowledge: Enhanced Multimodal Understanding and Recognition

TL;DR

This work analyzes core multimodal machine learning challenges—alignment, translation, fusion, and transference—across five chapters. It introduces Spatial-Reasoning Bert to translate language-based spatial relations into 2D layouts, and proposes Soft Organ Distance loss to ground medical text within a 3D human atlas, enabling interpretable, spatially-aware navigation of medical information. It also develops FaLB for translating structured text to canonical knowledge-graph facts and demonstrates multimodal fusion strategies for compositional action recognition, alongside a multimodal knowledge distillation approach to sustain performance with RGB-only inference in egocentric action recognition. Collectively, the study advances spatial-language understanding, medical-text grounding, knowledge-graph enrichment, and resource-efficient action recognition, with broad implications for automated scene generation, medical visualization, and robust video understanding. The work emphasizes maintaining modality complementarity, grounding to intuitive abstract spaces, and leveraging synthetic data to scale cross-modal linking tasks while addressing out-of-knowledge challenges.

Abstract

This manuscript explores multimodal alignment, translation, fusion, and transference to enhance machine understanding of complex inputs. We organize the work into five chapters, each addressing unique challenges in multimodal machine learning. Chapter 3 introduces Spatial-Reasoning Bert for translating text-based spatial relations into 2D arrangements between clip-arts. This enables effective decoding of spatial language into visual representations, paving the way for automated scene generation aligned with human spatial understanding. Chapter 4 presents a method for translating medical texts into specific 3D locations within an anatomical atlas. We introduce a loss function leveraging spatial co-occurrences of medical terms to create interpretable mappings, significantly enhancing medical text navigability. Chapter 5 tackles translating structured text into canonical facts within knowledge graphs. We develop a benchmark for linking natural language to entities and predicates, addressing ambiguities in text extraction to provide clearer, actionable insights. Chapter 6 explores multimodal fusion methods for compositional action recognition. We propose a method fusing video frames and object detection representations, improving recognition robustness and accuracy. Chapter 7 investigates multimodal knowledge transference for egocentric action recognition. We demonstrate how multimodal knowledge distillation enables RGB-only models to mimic multimodal fusion-based capabilities, reducing computational requirements while maintaining performance. These contributions advance methodologies for spatial language understanding, medical text interpretation, knowledge graph enrichment, and action recognition, enhancing computational systems' ability to process complex, multimodal inputs across diverse applications.

Figures (26)

• Figure 1: Abstract modalities used in Part \ref{['part:alignment-translation']} of this manuscript. (a) is used in Chapter \ref{['ch:emnlp2020']}; (b) is used in Chapter \ref{['ch:conll2020']}; (c) is used in Chapter \ref{['ch:emnlp2023']}.
• Figure 2: Video frames (represented as a raw modality), and the abstract modalities used in Part \ref{['part:fusion-transference']} of this manuscript.
• Figure 3: The Transformer encoder-decoder model architecture. Image source: Vaswani et al.vaswani2017attention. It is important to note that within the context of this manuscript, and the broader literature, references to the Transformer model typically pertain to the Transformer encoder only.
• Figure 4: Given a set of clip-arts and a textual description of a scene, including both implicit and explicit spatial relations, our method automatically generates a spatial arrangement of the clip-arts that satisfies the spatial constraints.
• Figure 5: The sr-bert sr-bert model with the text position embedding module as per Bert---Left (Yellow), clip-art spatial embedding module, which is novel in sr-bert sr-bert---Right (Blue). The blue [MASK] elements are the masked spatial positions, which the model learns to predict during training. During inference, all blue elements (the spatial encoding of the clip-arts) are masked, and the model non-autoregressively decodes them.