Describing Textures in the Wild

TL;DR

Addressing texture description in natural images, the paper introduces the Describable Textures Dataset (DTD) with 47 attributes collected in the wild. It shows that an Improved Fisher Vector (IFV) representation, originally for object recognition, is highly effective for texture attribute recognition, surpassing specialized texture descriptors and improving multiple material-recognition benchmarks. Moreover, the describable attributes themselves form a compact, transferable representation that enhances material classification when integrated with IFV. The work also demonstrates practical uses in search and visualization, enabling intuitive descriptions of materials in real-world images and catalogs.

Abstract

Patterns and textures are defining characteristics of many natural objects: a shirt can be striped, the wings of a butterfly can be veined, and the skin of an animal can be scaly. Aiming at supporting this analytical dimension in image understanding, we address the challenging problem of describing textures with semantic attributes. We identify a rich vocabulary of forty-seven texture terms and use them to describe a large dataset of patterns collected in the wild.The resulting Describable Textures Dataset (DTD) is the basis to seek for the best texture representation for recognizing describable texture attributes in images. We port from object recognition to texture recognition the Improved Fisher Vector (IFV) and show that, surprisingly, it outperforms specialized texture descriptors not only on our problem, but also in established material recognition datasets. We also show that the describable attributes are excellent texture descriptors, transferring between datasets and tasks; in particular, combined with IFV, they significantly outperform the state-of-the-art by more than 8 percent on both FMD and KTHTIPS-2b benchmarks. We also demonstrate that they produce intuitive descriptions of materials and Internet images.

Figures (12)

• Figure 1: Both the man-made and the natural world are an abundant source of richly textured objects. The textures of objects shown above can be described (in no particular order) as dotted, striped, chequered, cracked, swirly, honeycombed, and scaly. We aim at identifying these attributes automatically and generating descriptions based on them.
• Figure 2: The 47 texture words in the describable texture dataset introduced in this paper. Two examples of each attribute are shown to illustrate the significant amount of variability in the data.
• Figure 3: Quality of joint sequential annotations. Each bar shows the average number of occurrences of a given attribute in a DTD image. The horizontal dashed line corresponds to a frequency of 1/47, the minimum given the design of DTD (Sect. \ref{['s:design']}). The black portion of each bar is the amount of attributes discovered by the sequential procedure, using only 10 annotations per image (about one fifth of the effort required for exhaustive annotation). The orange portion shows the additional recall obtained by integrating CV in the process. Right: co-occurrence of attributes. The matrix shows the joint probability $p(q,q')$ of two attributes occurring together (rows and columns are sorted in the same way as the left image).
• Figure 4: Per-class AP of the 47 describable attribute classifiers on DTD using the $\text{IFV}_\text{SIFT}$ representation and linear classifiers.
• Figure 5: Descriptions of materials from KTH-TIPS-2b dataset. These words are the most frequent top scoring texture attributes (from the list of 47 we proposed), when classifying the images from the KTH-TIPS-2b dataset.