EL-VIT: Probing Vision Transformer with Interactive Visualization

TL;DR

The paper tackles the steep learning curve of Vision Transformers by introducing EL-VIT, a web-based, multi-view visualization system. EL-VIT employs four interconnected views—Model Overview, Knowledge Background Graph, Model Detail View, and Interpretation View—to explain architectural, mathematical, and interpretability aspects of ViT. A key novelty is the cosine-similarity interpretation that reveals patch-level relationships across Transformer blocks, offering an alternative to traditional attention-weight visualizations. The authors demonstrate the tool's educational value through two usage scenarios and provide a browser-accessible implementation built on TensorFlow.js with a ViT model fine-tuned on CIFAR-10 from Hugging Face.

Abstract

Nowadays, Vision Transformer (ViT) is widely utilized in various computer vision tasks, owing to its unique self-attention mechanism. However, the model architecture of ViT is complex and often challenging to comprehend, leading to a steep learning curve. ViT developers and users frequently encounter difficulties in interpreting its inner workings. Therefore, a visualization system is needed to assist ViT users in understanding its functionality. This paper introduces EL-VIT, an interactive visual analytics system designed to probe the Vision Transformer and facilitate a better understanding of its operations. The system consists of four layers of visualization views. The first three layers include model overview, knowledge background graph, and model detail view. These three layers elucidate the operation process of ViT from three perspectives: the overall model architecture, detailed explanation, and mathematical operations, enabling users to understand the underlying principles and the transition process between layers. The fourth interpretation view helps ViT users and experts gain a deeper understanding by calculating the cosine similarity between patches. Our two usage scenarios demonstrate the effectiveness and usability of EL-VIT in helping ViT users understand the working mechanism of ViT.

Figures (11)

• Figure 1: The overview diagram of the visualization system.
• Figure 2: The architectural diagram of the ViT-B/16 model.
• Figure 3: Model Overview. (A) illustrates the overall structure of the ViT model. (B) showcases the 12 layers of the Transformer encoder. (C) details the composition of each Transformer block at every layer.
• Figure 4: Knowledge Background Graph. (A) depicts as a force-directed graph, visualizing crucial concepts and the code's architecture. (B) displays the specific content of the node through modal box.
• Figure 5: Model Detail View. (A) depicts the overall model process of parameters visualization. (B) displays parameter visualizations of the outputs from 12 Transformer blocks. (C) illustrates parameter visualizations for each internal layer within a Transformer block. (D) visualizes parameters of the internal layers within the Multi-Head Attention layer.