The When and How of Target Variable Transformations

TL;DR

The paper argues that transforming the target variable, not just the input features, can substantially improve learnability in regression tasks. It formalizes the target transformation problem, reviews related work, and delivers a practical guide that categorizes transformations into subjective, contextual, and distribution-based approaches, with concrete rules of thumb. Through an empirical study across ten imbalanced datasets and four regression models, it shows that variance-stabilizing and distribution-shaping transforms (e.g., QuantileNormal, Yeo-Johnson, and logarithm) consistently improve performance for linear models and SVR, while tree-based methods are more robust. The work provides open-source code and actionable guidance, highlighting target transformations as a standard preprocessing tool with practical impact for real-world data.

Abstract

The machine learning pipeline typically involves the iterative process of (1) collecting the data, (2) preparing the data, (3) learning a model, and (4) evaluating a model. Practitioners recognize the importance of the data preparation phase in terms of its impact on the ability to learn accurate models. In this regard, significant attention is often paid to manipulating the feature set (e.g., selection, transformations, dimensionality reduction). A point that is less well appreciated is that transformations on the target variable can also have a large impact on whether it is possible to learn a suitable model. These transformations may include accounting for subject-specific biases (e.g., in how someone uses a rating scale), contexts (e.g., population size effects), and general trends (e.g., inflation). However, this point has received a much more cursory treatment in the existing literature. The goal of this paper is three-fold. First, we aim to highlight the importance of this problem by showing when transforming the target variable has been useful in practice. Second, we will provide a set of generic ``rules of thumb'' that indicate situations when transforming the target variable may be needed. Third, we will discuss which transformations should be considered in a given situation.