Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Intra-tree Column Subsampling Hinders XGBoost Learning of Ratio-like Interactions

Mykola Pinchuk

TL;DR

This paper addresses whether intra-tree column subsampling in XGBoost obstructs learning of ratio-like interactions that cancel a shared nuisance factor. It uses cancellation-style synthetic data with two primitives and an engineered ratio to quantify the impact across multiple subsampling settings, showing substantial PR-AUC degradation when only primitives are used and subsampling is aggressive. The engineered ratio feature largely mitigates the effect, highlighting a practical approach: include ratio-like features if plausible, or avoid intra-tree subsampling in such scenarios. The findings provide actionable guidance for practitioners balancing speed, regularization, and the need to capture ratio-like signals in imbalanced tasks.

Abstract

Many applied problems contain signal that becomes clear only after combining multiple raw measurements. Ratios and rates are common examples. In gradient boosted trees, this combination is not an explicit operation: the model must synthesize it through coordinated splits on the component features. We study whether intra-tree column subsampling in XGBoost makes that synthesis harder. We use two synthetic data generating processes with cancellation-style structure. In both, two primitive features share a strong nuisance factor, while the target depends on a smaller differential factor. A log ratio cancels the nuisance and isolates the signal. We vary colsample_bylevel and colsample_bynode over s in {0.4, 0.6, 0.8, 0.9}, emphasizing mild subsampling (s >= 0.8). A control feature set includes the engineered ratio, removing the need for synthesis. Across both processes, intra-tree column subsampling reduces test PR-AUC in the primitives-only setting. In the main process the relative decrease reaches 54 percent when both parameters are set to 0.4. The effect largely disappears when the engineered ratio is present. A path-based co-usage metric drops in the same cells where performance deteriorates. Practically, if ratio-like structure is plausible, either avoid intra-tree subsampling or include the intended ratio features.

Intra-tree Column Subsampling Hinders XGBoost Learning of Ratio-like Interactions

TL;DR

This paper addresses whether intra-tree column subsampling in XGBoost obstructs learning of ratio-like interactions that cancel a shared nuisance factor. It uses cancellation-style synthetic data with two primitives and an engineered ratio to quantify the impact across multiple subsampling settings, showing substantial PR-AUC degradation when only primitives are used and subsampling is aggressive. The engineered ratio feature largely mitigates the effect, highlighting a practical approach: include ratio-like features if plausible, or avoid intra-tree subsampling in such scenarios. The findings provide actionable guidance for practitioners balancing speed, regularization, and the need to capture ratio-like signals in imbalanced tasks.

Abstract

Many applied problems contain signal that becomes clear only after combining multiple raw measurements. Ratios and rates are common examples. In gradient boosted trees, this combination is not an explicit operation: the model must synthesize it through coordinated splits on the component features. We study whether intra-tree column subsampling in XGBoost makes that synthesis harder. We use two synthetic data generating processes with cancellation-style structure. In both, two primitive features share a strong nuisance factor, while the target depends on a smaller differential factor. A log ratio cancels the nuisance and isolates the signal. We vary colsample_bylevel and colsample_bynode over s in {0.4, 0.6, 0.8, 0.9}, emphasizing mild subsampling (s >= 0.8). A control feature set includes the engineered ratio, removing the need for synthesis. Across both processes, intra-tree column subsampling reduces test PR-AUC in the primitives-only setting. In the main process the relative decrease reaches 54 percent when both parameters are set to 0.4. The effect largely disappears when the engineered ratio is present. A path-based co-usage metric drops in the same cells where performance deteriorates. Practically, if ratio-like structure is plausible, either avoid intra-tree subsampling or include the intended ratio features.
Paper Structure (22 sections, 10 equations, 7 figures, 2 tables)

This paper contains 22 sections, 10 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Column subsampling in XGBoost
  4. Related work
  5. Experimental setup
  6. Feature sets
  7. Column subsampling arms
  8. Data generating processes
  9. Metrics
  10. Results
  11. Discussion
  12. What these results do and do not say
  13. Practical guidance
  14. Limitations and next steps
  15. Conclusion
  16. ...and 7 more sections

Figures (7)

  • Figure 1: DGP-A (primitives only, F0). Relative change in test PR-AUC (left axis) and change in path co-usage (right axis) as a function of the within-tree sampling rate $s$ for arms C1--C3.
  • Figure 2: DGP-A (F0). Relative $\Delta$ROC-AUC (left axis) and $\Delta$cooc_path_mean (right axis) across $s$ and arms C1--C3.
  • Figure 3: DGP-B (F0). Relative $\Delta$PR-AUC (left axis) and $\Delta$cooc_path_mean (right axis) across $s$ and arms C1--C3.
  • Figure 4: DGP-B (F0). Relative $\Delta$ROC-AUC (left axis) and $\Delta$cooc_path_mean (right axis) across $s$ and arms C1--C3.
  • Figure 5: DGP-A low $s$ sweep. The sweep includes $s \in \{0.4,0.6,0.8,0.9\}$. The interaction synthesis setting F0 is more sensitive to stronger intra-tree subsampling.
  • ...and 2 more figures