Omnivore: A Single Model for Many Visual Modalities

TL;DR

Omnivore addresses modality fragmentation by training a single transformer-based vision system that operates across images, videos, and single-view 3D data using shared representations. It converts all inputs into a common patch-based embedding and processes them with a Swin Transformer, enabling cross-modal generalization without paired data. Empirically, Omnivore matches or surpasses modality-specific baselines and achieves competitive state-of-the-art results across ImageNet, Kinetics, EPIC-Kitchens, SSv2, and NYU RGB-D in transfer settings. This work suggests that a unified, cross-modality vision model can leverage diverse datasets to deliver strong performance with reduced architecture specialization and simpler deployment across modalities.

Abstract

Prior work has studied different visual modalities in isolation and developed separate architectures for recognition of images, videos, and 3D data. Instead, in this paper, we propose a single model which excels at classifying images, videos, and single-view 3D data using exactly the same model parameters. Our 'Omnivore' model leverages the flexibility of transformer-based architectures and is trained jointly on classification tasks from different modalities. Omnivore is simple to train, uses off-the-shelf standard datasets, and performs at-par or better than modality-specific models of the same size. A single Omnivore model obtains 86.0% on ImageNet, 84.1% on Kinetics, and 67.1% on SUN RGB-D. After finetuning, our models outperform prior work on a variety of vision tasks and generalize across modalities. Omnivore's shared visual representation naturally enables cross-modal recognition without access to correspondences between modalities. We hope our results motivate researchers to model visual modalities together.

Figures (7)

• Figure 1: Omnivore is a single vision model for many different visual modalities. It learns to construct representations that are aligned across visual modalities, without requiring training data that specifies correspondences between those modalities. Using Omnivore's shared visual representation, we successfully identify nearest neighbors of left: an image (ImageNet-1K validation set) in vision datasets that contain right: depth maps (ImageNet-1K training set), single-view 3D images (ImageNet-1K training set), and videos (Kinetics-400 validation set).
• Figure 2: Multiple visual modalities in the Omnivore model. We convert image, video, and single-view 3D modalities into embeddings that are fed into a Transformer model. The images are converted into patches, videos into spatio-temporal tubes, and the single-view 3D images are converted into RGB patches and depth patches. The patches are projected into embeddings using linear layers. We use the same linear layer for (image or video) RGB patches and a separate one for depth patches.
• Figure 3: Comparing Omnivore with VideoSwin on K400. Omnivore improves over VideoSwin on F1 score on all 38 class groups defined in kay2017kinetics (top 15 shown here for brevity).
• Figure 4: Retrieving depth maps given RGB images on the ImageNet-1K dataset. We show retrieved depth maps from the IN1K training set (right) for RGB image queries from the IN1K validation set (left). Although Omnivore was not trained on IN1K depth maps, the shared visual representation enables it to retrieve depth maps that are semantically similar to the query.
• Figure 5: Accuracy as a function of clip length on the K400 dataset. Models are trained on 32-frame clips but evaluated on clips of different length (with the same fps used for frame sampling). The performance of Omnivore degrades more gracefully than that of the VideoSwin-B model, and is still effective when doing frame-level inference (i.e., when the clip length is 1).