Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A hierarchical loss and its problems when classifying non-hierarchically

Cinna Wu, Mark Tygert, Yann LeCun

TL;DR

This work defines a metric that, inter alia, can penalize failure to distinguish between a sheepdog and a skyscraper more thanfailure to distinguish Between a sheepdogs and a poodles.

Abstract

Failing to distinguish between a sheepdog and a skyscraper should be worse and penalized more than failing to distinguish between a sheepdog and a poodle; after all, sheepdogs and poodles are both breeds of dogs. However, existing metrics of failure (so-called "loss" or "win") used in textual or visual classification/recognition via neural networks seldom leverage a-priori information, such as a sheepdog being more similar to a poodle than to a skyscraper. We define a metric that, inter alia, can penalize failure to distinguish between a sheepdog and a skyscraper more than failure to distinguish between a sheepdog and a poodle. Unlike previously employed possibilities, this metric is based on an ultrametric tree associated with any given tree organization into a semantically meaningful hierarchy of a classifier's classes. An ultrametric tree is a tree with a so-called ultrametric distance metric such that all leaves are at the same distance from the root. Unfortunately, extensive numerical experiments indicate that the standard practice of training neural networks via stochastic gradient descent with random starting points often drives down the hierarchical loss nearly as much when minimizing the standard cross-entropy loss as when trying to minimize the hierarchical loss directly. Thus, this hierarchical loss is unreliable as an objective for plain, randomly started stochastic gradient descent to minimize; the main value of the hierarchical loss may be merely as a meaningful metric of success of a classifier.

A hierarchical loss and its problems when classifying non-hierarchically

TL;DR

This work defines a metric that, inter alia, can penalize failure to distinguish between a sheepdog and a skyscraper more thanfailure to distinguish Between a sheepdogs and a poodles.

Abstract

Failing to distinguish between a sheepdog and a skyscraper should be worse and penalized more than failing to distinguish between a sheepdog and a poodle; after all, sheepdogs and poodles are both breeds of dogs. However, existing metrics of failure (so-called "loss" or "win") used in textual or visual classification/recognition via neural networks seldom leverage a-priori information, such as a sheepdog being more similar to a poodle than to a skyscraper. We define a metric that, inter alia, can penalize failure to distinguish between a sheepdog and a skyscraper more than failure to distinguish between a sheepdog and a poodle. Unlike previously employed possibilities, this metric is based on an ultrametric tree associated with any given tree organization into a semantically meaningful hierarchy of a classifier's classes. An ultrametric tree is a tree with a so-called ultrametric distance metric such that all leaves are at the same distance from the root. Unfortunately, extensive numerical experiments indicate that the standard practice of training neural networks via stochastic gradient descent with random starting points often drives down the hierarchical loss nearly as much when minimizing the standard cross-entropy loss as when trying to minimize the hierarchical loss directly. Thus, this hierarchical loss is unreliable as an objective for plain, randomly started stochastic gradient descent to minimize; the main value of the hierarchical loss may be merely as a meaningful metric of success of a classifier.

Paper Structure

This paper contains 19 sections, 4 figures, 7 tables.

Table of Contents

  1. Abstract
  2. Introduction
  3. Methods
  4. Results and discussion
  5. Conclusion
  6. Acknowledgements

Figures (4)

  • Figure 1: This is a hierarchy of types of fish (but not the whole hierarchy from Subsection \ref{['fishsec']} of Section \ref{['numex']}).
  • Figure 2: This depicts the hierarchy of Fig \ref{['fishhier']} when the correct target leaf is "acipenser (oxyrinchus)" but the leaf predicted with probability 1 (the "single best class" of Subsection \ref{['inbest']}) is "huso" (as indicated with a box around "huso"). In this case, the hierarchical win is 1/2 + 1/4 = 3/4, as "acipenseriformes" and "sturgeon" are the nodes on both the path from the root "acipenseriformes" to "acipenser (oxyrinchus)" and the path from the root "acipenseriformes" to "huso."
  • Figure 3: This depicts the hierarchy of Fig \ref{['fishhier']} when the correct target leaf is "acipenser (oxyrinchus)" but the leaf predicted with probability 1 (the "single best class" of Subsection \ref{['inbest']}) is "acipenser (other)" (as indicated with a box around "acipenser (other)"). In this case, the hierarchical win is 1/2 + 1/4 + 1/8 = 7/8, as "acipenseriformes," "sturgeon," and "acipenser" are the nodes on both the path from the root "acipenseriformes" to "acipenser (oxyrinchus)" and the path from the root "acipenseriformes" to "acipenser (other)."
  • Figure 4: This depicts the hierarchy of Fig \ref{['fishhier']} when the correct target leaf is "acipenser (oxyrinchus)," the same as the leaf predicted with probability 1 (the "single best class" of Subsection \ref{['inbest']}). In this case, the hierarchical win is 1/2 + 1/4 + 1/8 + 1/8 = 1 (as with all correct predictions); the final node is double-counted at 1/8 = 1/16 + 1/16.