Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring space efficiency in a tree-based linear model for extreme multi-label classification

He-Zhe Lin, Cheng-Hung Liu, Chih-Jen Lin

TL;DR

This research provides a simple procedure to estimate the size of a tree model before training any classifier in the tree nodes, and if the model size is already acceptable, this approach can help avoid modifying the model through weight pruning or other techniques.

Abstract

Extreme multi-label classification (XMC) aims to identify relevant subsets from numerous labels. Among the various approaches for XMC, tree-based linear models are effective due to their superior efficiency and simplicity. However, the space complexity of tree-based methods is not well-studied. Many past works assume that storing the model is not affordable and apply techniques such as pruning to save space, which may lead to performance loss. In this work, we conduct both theoretical and empirical analyses on the space to store a tree model under the assumption of sparse data, a condition frequently met in text data. We found that, some features may be unused when training binary classifiers in a tree method, resulting in zero values in the weight vectors. Hence, storing only non-zero elements can greatly save space. Our experimental results indicate that tree models can achieve up to a 95% reduction in storage space compared to the standard one-vs-rest method for multi-label text classification. Our research provides a simple procedure to estimate the size of a tree model before training any classifier in the tree nodes. Then, if the model size is already acceptable, this approach can help avoid modifying the model through weight pruning or other techniques.

Exploring space efficiency in a tree-based linear model for extreme multi-label classification

TL;DR

This research provides a simple procedure to estimate the size of a tree model before training any classifier in the tree nodes, and if the model size is already acceptable, this approach can help avoid modifying the model through weight pruning or other techniques.

Abstract

Extreme multi-label classification (XMC) aims to identify relevant subsets from numerous labels. Among the various approaches for XMC, tree-based linear models are effective due to their superior efficiency and simplicity. However, the space complexity of tree-based methods is not well-studied. Many past works assume that storing the model is not affordable and apply techniques such as pruning to save space, which may lead to performance loss. In this work, we conduct both theoretical and empirical analyses on the space to store a tree model under the assumption of sparse data, a condition frequently met in text data. We found that, some features may be unused when training binary classifiers in a tree method, resulting in zero values in the weight vectors. Hence, storing only non-zero elements can greatly save space. Our experimental results indicate that tree models can achieve up to a 95% reduction in storage space compared to the standard one-vs-rest method for multi-label text classification. Our research provides a simple procedure to estimate the size of a tree model before training any classifier in the tree nodes. Then, if the model size is already acceptable, this approach can help avoid modifying the model through weight pruning or other techniques.

Paper Structure

This paper contains 24 sections, 3 theorems, 38 equations, 8 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Linear Methods for XMC
  3. One-vs-rest (OVR) Method
  4. Tree-based Methods
  5. Time and Space Analysis for Linear Methods
  6. Time Analysis
  7. Space Analysis
  8. Techniques for Reducing Model-size and Their Issues
  9. Inherent Pruning in Tree-based Methods for Sparse Data
  10. Analysis on Balanced Trees
  11. Experimental Results
  12. Experimental Settings
  13. Estimating Model Size Prior to Training
  14. Empirical Analysis on the Model Size
  15. Empirical Study on the Reduction Rate $\alpha$
  16. ...and 9 more sections

Key Result

Theorem 1

Consider $2 < d \le D$ and assume $K \ge 4$. Let $\alpha^*$ be the unique solution in $(0, 1)$ of the equation If $\alpha < \max\{2/K, \alpha^*\}$, then the ratio eq:balanced-nnz-ratio is smaller than one.

Figures (8)

  • Figure 1: A label tree with nine labels. We set the number of clusters $K=3$ at each node for the label partition. In the figure, each internal node colored red is associated with a label subset, and each leaf node colored blue corresponds to a single label.
  • Figure 2: Illustration for a depth $d$ balanced label tree with number of clusters $K$. The number within $(\cdot)$ at each node means the size of the node's label subset. Nodes from depth $0$ to depth $(d-2)$ all have $K$ children. Because the tree needs to be terminated at depth $d$, each node at depth $d-1$ has $L/K^{d-1}$ children.
  • Figure 3: The ratio of number of non-zeros between a tree model and an OVR model, calculated based on \ref{['eq:balanced-nnz-ratio']}. We show the cases for $\alpha=\{0.3, 0.4, 0.5, 0.6\}$.
  • Figure 4: The ratio between the tree model size and the OVR model size with $K=100$ and various $d_\text{max}$. The box in each sub-figure shows the actual model size of tree/OVR models under $d_\text{max} = 6$ and the ratio between the two.
  • Figure 5: The ratio between the tree model size and the OVR model size with $d_\text{max} = \{2,3,4\}$ and $K=\lceil L^{1/d_\text{max}}\rceil$.
  • ...and 3 more figures

Theorems & Definitions (5)

  • Theorem 1
  • Theorem 2
  • proof : Proof of Theorem \ref{['thm:ratio-smaller-than-1']}
  • Theorem 3
  • proof : Proof of Theorem \ref{['thm:lower-bound-of-alpha']}