Extracting Symbolic Sequences from Visual Representations via Self-Supervised Learning

TL;DR

This work investigates extracting discrete symbolic sequences from visual data using self-supervised learning, aiming to bridge continuous visual representations with symbolic reasoning. A teacher–student framework extends DINO, where a frozen ViT-based teacher guides a decoder-based student to generate and discretize symbolic tokens that can be linked to image regions through cross-attention. The study demonstrates that the generated symbolic representations capture meaningful abstraction, with interpretability supported by attention maps and compositional subsequences improving detail capture, though there are limitations in discretization and data scale. Overall, the approach lays groundwork for interpretable, high-level visual understanding via symbolic reasoning and points to scalable extensions on larger datasets and refined discretization methods.

Abstract

This paper explores the potential of abstracting complex visual information into discrete, structured symbolic sequences using self-supervised learning (SSL). Inspired by how language abstracts and organizes information to enable better reasoning and generalization, we propose a novel approach for generating symbolic representations from visual data. To learn these sequences, we extend the DINO framework to handle visual and symbolic information. Initial experiments suggest that the generated symbolic sequences capture a meaningful level of abstraction, though further refinement is required. An advantage of our method is its interpretability: the sequences are produced by a decoder transformer using cross-attention, allowing attention maps to be linked to specific symbols and offering insight into how these representations correspond to image regions. This approach lays the foundation for creating interpretable symbolic representations with potential applications in high-level scene understanding.

Figures (5)

• Figure 1: Visualization of four sample images alongside their generated sequences and corresponding attention masks, produced by our model. The sequences are generated using a temperature-softmax discretization process with a temperature of 0.12 during training. Attention masks, associated with each sequence element, are extracted from the cross-attention layers in the deepest layer of the descriptor module. From left to right: The first column shows the input sample images, followed by the generated sequences and their corresponding attention masks.
• Figure 2: Schematic drawing of the teacher-student setup. The teacher model consists as usual of an encoder and projector, while the student models consist of a decoder and encoder plus the regular projector. The input images are passed to a pretrained teacher, and the representations of it are then fed to the student. finally, the outputs of the projectors are compared. The student weights are then adjusted to mimic the output of the teacher. In our experiments, we work under the assumption of an existing visual encoder and focus solely on training the projector layer of the teacher using EMA while keeping the rest of it frozen.
• Figure 3: Training process of nine variations of our method, including three discretization variations with varied vocabulary sizes in symbolic descriptions. From left to right: (a) shows the teacher entropy over training steps; (b) displays the KL divergence between teacher and student distributions; (c) presents the evaluation performance using a k-NN metric across the different variations.
• Figure 4: Training curves for three exploration strategies: (a) Base Strategy, (b) Entropy Encouragement Strategy, and (c) Information Maximization Strategy. Each plot tracks multiple metrics over training steps: top-1 and top-5 classification accuracy (probing), training loss, teacher entropy, KL divergence between teacher and student distributions, information content of the generated sequences, and entropy of the logits from the decoder transformer. These metrics reflect the effects of the different strategies on exploration, variability, and performance during the symbolic sequence generation process.
• Figure 5: Qualitative analysis of symbolic interpretability for the "bird" class, focusing on the appearance of symbol 279 across multiple samples. The figure shows input images, followed by attention maps that highlight regions corresponding to symbol 279. This symbol consistently appears in specific locations across different samples, often linked to parts of the bird, such as the body. Such patterns are common in classes with low visual variability, like birds, whereas classes with higher variability (e.g., ships, not shown) exhibit more localized and less consistent behavior. All sequences and attention maps are generated using a temperature-softmax discretization with a temperature of 0.12.