Universal representations:The missing link between faces, text, planktons, and cat breeds

TL;DR

The paper investigates whether neural networks can learn universal representations that transfer across highly diverse visual domains by training a single network on multiple tasks. It demonstrates that substantial sharing is feasible, even across domains as different as digits, faces, objects, and text, provided normalization is carefully managed—either via domain-specific scaling or through instance normalization that supports a universal parameter set. Across small and large datasets, deep sharing often outperforms separate models while reducing parameters, highlighting a shared visual representation underlying multiple tasks. These findings suggest universal representations are within reach of current architectures, given appropriate capacity and normalization strategies, with practical implications for building more general-purpose vision systems.

Abstract

With the advent of large labelled datasets and high-capacity models, the performance of machine vision systems has been improving rapidly. However, the technology has still major limitations, starting from the fact that different vision problems are still solved by different models, trained from scratch or fine-tuned on the target data. The human visual system, in stark contrast, learns a universal representation for vision in the early life of an individual. This representation works well for an enormous variety of vision problems, with little or no change, with the major advantage of requiring little training data to solve any of them. In this paper we investigate whether neural networks may work as universal representations by studying their capacity in relation to the “size” of a large combination of vision problems. We do so by showing that a single neural network can learn simultaneously several very different visual domains (from sketches to planktons and MNIST digits) as well as, or better than, a number of specialized networks. However, we also show that this requires to carefully normalize the information in the network, by using domain-specific scaling factors or, more generically, by using an instance normalization layer.

Figures (4)

• Figure 1: Humans posses an internal visual representation that, out of the box, works very well any number of visual domains, from objects and faces to planktons and characters. In this paper we investigate such universal representations by constructing neural networks that work simultaneously on many domains, learning to share common visual structure where no obvious commonality exists. Our goal is to contrast the capacity of such model against the total size of the combined vision problems.
• Figure 2: From left to right, three example modules: instance normalization, batch normalization, and batch normalization with domain-specific scaling building modules. The shaded blocks indicate learnable parameters. Other variants are tested, not shown for compactness.
• Figure 3: Example images from various datasets.
• Figure 4: Example images from the large-scale datasets are shown in their relative sizes.