Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Collapsing Categories for Regression with Mixed Predictors

Chaegeun Song, Zhong Zheng, Bing Li, Lingzhou Xue

TL;DR

This work tackles regression with mixed predictors by adaptively collapsing categorical levels through a pairwise vector fused LASSO (PVF-LASSO). By formulating a general loss L(beta) that covers linear models and GLMs, and by proving category collapsing consistency under an irrepresentable-type condition, the authors provide both theoretical guarantees and a scalable algorithm—the Inexact Proximal Gradient (IPG) with dual BD subproblem—that reliably recovers true category groupings. The adaptive extension weights penalties to improve selection and estimation, and the method is demonstrated to reduce categorical complexity while boosting predictive performance in simulations and in a Spotify song popularity dataset. The approach offers a principled, data-driven mechanism to simplify regression models with mixed predictors, with potential broad impact on applied statistics and machine learning tasks where categorical structure is rich and high-dimensional.

Abstract

Categorical predictors are omnipresent in everyday regression practice: in fact, most regression data involve some categorical predictors, and this tendency is increasing in modern applications with more complex structures and larger data sizes. However, including too many categories in a regression model would seriously hamper accuracy, as the information in the data is fragmented by the multitude of categories. In this paper, we introduce a systematic method to reduce the complexity of categorical predictors by adaptively collapsing categories in regressions, so as to enhance the performance of regression estimation. Our method is based on the {\em pairwise vector fused LASSO}, which automatically fuses the categories that bear a similar regression relation with the response. We develop our method under a wide class of regression models defined by a general loss function, which includes linear models and generalized linear models as special cases. We rigorously established the category collapsing consistency of our method, developed an Inexact Proximal Gradient Descent algorithm to implement it, and proved the feasibility and convergence of our algorithm. Through simulations and an application to Spotify music data, we demonstrate that our method can effectively reduce categorical complexity while improving prediction performance, making it a powerful tool for regression with mixed predictors.

Collapsing Categories for Regression with Mixed Predictors

TL;DR

This work tackles regression with mixed predictors by adaptively collapsing categorical levels through a pairwise vector fused LASSO (PVF-LASSO). By formulating a general loss L(beta) that covers linear models and GLMs, and by proving category collapsing consistency under an irrepresentable-type condition, the authors provide both theoretical guarantees and a scalable algorithm—the Inexact Proximal Gradient (IPG) with dual BD subproblem—that reliably recovers true category groupings. The adaptive extension weights penalties to improve selection and estimation, and the method is demonstrated to reduce categorical complexity while boosting predictive performance in simulations and in a Spotify song popularity dataset. The approach offers a principled, data-driven mechanism to simplify regression models with mixed predictors, with potential broad impact on applied statistics and machine learning tasks where categorical structure is rich and high-dimensional.

Abstract

Categorical predictors are omnipresent in everyday regression practice: in fact, most regression data involve some categorical predictors, and this tendency is increasing in modern applications with more complex structures and larger data sizes. However, including too many categories in a regression model would seriously hamper accuracy, as the information in the data is fragmented by the multitude of categories. In this paper, we introduce a systematic method to reduce the complexity of categorical predictors by adaptively collapsing categories in regressions, so as to enhance the performance of regression estimation. Our method is based on the {\em pairwise vector fused LASSO}, which automatically fuses the categories that bear a similar regression relation with the response. We develop our method under a wide class of regression models defined by a general loss function, which includes linear models and generalized linear models as special cases. We rigorously established the category collapsing consistency of our method, developed an Inexact Proximal Gradient Descent algorithm to implement it, and proved the feasibility and convergence of our algorithm. Through simulations and an application to Spotify music data, we demonstrate that our method can effectively reduce categorical complexity while improving prediction performance, making it a powerful tool for regression with mixed predictors.

Paper Structure

This paper contains 15 sections, 12 theorems, 54 equations, 2 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Pairwise vector fused LASSO for linear regression
  3. The procedure
  4. Definition of category collapsing consistency
  5. Irrepresentable condition
  6. Establishing category collapsing consistency
  7. Category collapsing for general regression
  8. General regression
  9. Generalized linear models
  10. Adaptive pairwise vector fused LASSO
  11. Optimization
  12. Inexact proximal gradient algorithm for linear regression
  13. Inexact proximal gradient algorithm for general regression
  14. Simulation
  15. Spotify music data

Key Result

Lemma 2.1

Condition (eq:category collapsing consistency: original) is equivalent to

Figures (2)

  • Figure 1: Solution paths for the estimated pairwise differences of the coefficients in the pairwise vector fused LASSO and adaptive methods for Scenarios 1 and 2. The top panels correspond to Scenario 1, and the bottom panels represent Scenario 2. Left Panels show the results for the pairwise vector fused LASSO, and the Right panels show the results for the adaptive methods.
  • Figure 2: The pairwise $\ell_2$-distances between the coefficient vectors of each subgenre

Theorems & Definitions (12)

  • Lemma 2.1
  • Proposition 2.1
  • Lemma 2.2
  • Lemma 2.3
  • Lemma 2.4
  • Lemma 2.5
  • Lemma 2.6
  • Theorem 2.7
  • Theorem 3.1
  • Theorem 4.1
  • ...and 2 more