An Efficient and Explanatory Image and Text Clustering System with Multimodal Autoencoder Architecture

TL;DR

The paper introduces CRVAE, a dense, multimodal variational autoencoder that fuses CNN-derived image features with LSTM-derived text to produce a compact latent representation for video frames and captions. A largely automatic pipeline clusters latent vectors with K-means and leverages BLIP captions and LLaMA-derived tags, augmented by PhraseBERT embeddings, to provide human-interpretable, cross-cultural cluster explanations. The approach is validated on English and Chinese COVID-19 and Winter Olympics videos, showing clearer reconstructions, meaningful clusters, and insightful cross-cultural similarities and differences, while noting some hallucinations in generated tags. The work demonstrates practical, scalable cross-cultural video analysis and explainability, with potential extensions to transformers and broader topic coverage for rapid comparative media studies.

Abstract

We demonstrate the efficiencies and explanatory abilities of extensions to the common tools of Autoencoders and LLM interpreters, in the novel context of comparing different cultural approaches to the same international news event. We develop a new Convolutional-Recurrent Variational Autoencoder (CRVAE) model that extends the modalities of previous CVAE models, by using fully-connected latent layers to embed in parallel the CNN encodings of video frames, together with the LSTM encodings of their related text derived from audio. We incorporate the model within a larger system that includes frame-caption alignment, latent space vector clustering, and a novel LLM-based cluster interpreter. We measure, tune, and apply this system to the task of summarizing a video into three to five thematic clusters, with each theme described by ten LLM-produced phrases. We apply this system to two news topics, COVID-19 and the Winter Olympics, and five other topics are in progress.

Figures (9)

• Figure 1: Pure and dense CVAE Architectures.
• Figure 2: System Overview: CRVAE model encodes images and texts into a low-dimension latent space, the vector representations are clustered using $K$-means, and BLIP describes the video frames to help LLaMA summarize the frames and captions.
• Figure 3: CRVAE Architecture: the encoder, in parallel, uses CNN layers to encode images and uses RNN layers to encode texts. These are fused into fully-connected layers with shrinking size. The decoder has a symmetric structure.
• Figure 4: Sample frames extracted.
• Figure 5: Sample reconstructed images, COVID-19 data.