Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhance and Reuse: A Dual-Mechanism Approach to Boost Deep Forest for Label Distribution Learning

Jia-Le Xu, Shen-Huan Lyu, Yu-Nian Wang, Ning Chen, Zhihao Qu, Bin Tang, Baoliu Ye

TL;DR

This work addresses label distribution learning (LDL) by integrating a dual-mechanism extension to Deep Forest (DF). The Enhanced and Reused Feature Deep Forest (ERDF) introduces a feature enhancement module that explicitly leverages label correlations via PCA-derived relationship patterns and an input-feature mapping, plus a measure-aware feature reuse module that stabilizes training by selectively reusing lower-quality features. Ablation and benchmark studies on five LDL datasets demonstrate that both components are essential, with ERDF achieving superior performance across six metrics and showing robustness to hyperparameters. The approach offers a practical, scalable pathway to exploit label correlations in non-backpropagation-based deep ensembles and suggests broader applicability to other label-dependent tasks.

Abstract

Label distribution learning (LDL) requires the learner to predict the degree of correlation between each sample and each label. To achieve this, a crucial task during learning is to leverage the correlation among labels. Deep Forest (DF) is a deep learning framework based on tree ensembles, whose training phase does not rely on backpropagation. DF performs in-model feature transform using the prediction of each layer and achieves competitive performance on many tasks. However, its exploration in the field of LDL is still in its infancy. The few existing methods that apply DF to the field of LDL do not have effective ways to utilize the correlation among labels. Therefore, we propose a method named Enhanced and Reused Feature Deep Forest (ERDF). It mainly contains two mechanisms: feature enhancement exploiting label correlation and measure-aware feature reuse. The first one is to utilize the correlation among labels to enhance the original features, enabling the samples to acquire more comprehensive information for the task of LDL. The second one performs a reuse operation on the features of samples that perform worse than the previous layer on the validation set, in order to ensure the stability of the training process. This kind of Enhance-Reuse pattern not only enables samples to enrich their features but also validates the effectiveness of their new features and conducts a reuse process to prevent the noise from spreading further. Experiments show that our method outperforms other comparison algorithms on six evaluation metrics.

Enhance and Reuse: A Dual-Mechanism Approach to Boost Deep Forest for Label Distribution Learning

TL;DR

This work addresses label distribution learning (LDL) by integrating a dual-mechanism extension to Deep Forest (DF). The Enhanced and Reused Feature Deep Forest (ERDF) introduces a feature enhancement module that explicitly leverages label correlations via PCA-derived relationship patterns and an input-feature mapping, plus a measure-aware feature reuse module that stabilizes training by selectively reusing lower-quality features. Ablation and benchmark studies on five LDL datasets demonstrate that both components are essential, with ERDF achieving superior performance across six metrics and showing robustness to hyperparameters. The approach offers a practical, scalable pathway to exploit label correlations in non-backpropagation-based deep ensembles and suggests broader applicability to other label-dependent tasks.

Abstract

Label distribution learning (LDL) requires the learner to predict the degree of correlation between each sample and each label. To achieve this, a crucial task during learning is to leverage the correlation among labels. Deep Forest (DF) is a deep learning framework based on tree ensembles, whose training phase does not rely on backpropagation. DF performs in-model feature transform using the prediction of each layer and achieves competitive performance on many tasks. However, its exploration in the field of LDL is still in its infancy. The few existing methods that apply DF to the field of LDL do not have effective ways to utilize the correlation among labels. Therefore, we propose a method named Enhanced and Reused Feature Deep Forest (ERDF). It mainly contains two mechanisms: feature enhancement exploiting label correlation and measure-aware feature reuse. The first one is to utilize the correlation among labels to enhance the original features, enabling the samples to acquire more comprehensive information for the task of LDL. The second one performs a reuse operation on the features of samples that perform worse than the previous layer on the validation set, in order to ensure the stability of the training process. This kind of Enhance-Reuse pattern not only enables samples to enrich their features but also validates the effectiveness of their new features and conducts a reuse process to prevent the noise from spreading further. Experiments show that our method outperforms other comparison algorithms on six evaluation metrics.
Paper Structure (14 sections, 5 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 14 sections, 5 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary
  3. The Proposed Method
  4. Feature enhancement exploiting label correlation
  5. Measure-aware feature reuse
  6. The framework
  7. Experiments
  8. Dataset and configuration
  9. Benchmark experiment
  10. Ablation study
  11. Layer-wise Dynamics and Mechanism Analysis
  12. Interplay between feature enhancement and layer structure
  13. Interplay between feature reuse and layer structure
  14. Conclusion

Figures (4)

  • Figure 1: This is an example sample of label distribution learning. On the left is a landscape picture as an input, which contains multiple elements. On the right is the degree of relevance of each element to the picture, which together form the label of this picture: $\bm{d}_{\text{image}}=\{0.03, 0.24, 0.21, 0.4, 0.03, 0.09\}$. Mountain and water tend to appear together, so the values of these two dimensions are both large. When cloud are present, the values describing the degree of the sky tend to be large. These are all manifestations of the correlations between the labels.
  • Figure 2: This figure clearly presents the overall framework of ERDF. Each layer is composed of RF and ERF. After the training of the $l$-th layer, the first part of the new features $\bm{H}_l$ is obtained. These new features are composed of the validation prediction results of all forests in this layer. Then, the feature enhancement mechanism is conducted to obtain the second part of the new features $\bm{E}_l$. These two parts are concatenated to form $\bm{F}_{new}^{(l)}$, which is input into the feature reuse module. After reuse according to $\bm{G}_{l-1}$, the final new features $\bm{G}_l$ of this layer are obtained and concatenated with the original features before being passed to the next layer.
  • Figure 3: Visualization of the feature enhancement dynamics across layers on Movie and SBU. Fig. \ref{['fig:movie_heatmap']} and \ref{['fig:SBU_heatmap']} are heatmaps showing the enhanced feature correlation between layers. Fig. \ref{['fig:movie_radar']} and \ref{['fig:SBU_radar']} are radar charts, showing the average error between the predicted values of the enhancers and the ideal values in each dimension.
  • Figure 4: Comparison of KL divergence trajectory between ERDF (blue) and ERDF w/o fr (green) on dataset Movie and SBU. The left column is the training phase, and the right is the testing phase. The red star indicates the optimal layer during the training phase, which is the layer that gives the final prediction in the testing phase.