Neural Multivariate Regression: Qualitative Insights from the Unconstrained Feature Model

TL;DR

This work uses the Unconstrained Feature Model (UFM) to derive closed-form, qualitative insights into neural multivariate regression. It shows that, under the UFM, a single multi-task model attains strictly lower training MSE than multiple single-task models when the latter are equally or more regularized, and that multi-task training shares regularization benefits with target whitening/normalization. The authors further demonstrate that whitening (ZCA) and normalization reduce training MSE when the average target variance is below one and may hurt when it is above one, with empirical results on four datasets (three MuJoCo robotics tasks and CARLA 2D) confirming these predictions. Overall, the UFM provides a tractable bridge between theory and practice, guiding DNN design choices and data pre-processing strategies for multivariate regression and imitation/reinforcement-learning contexts.

Abstract

The Unconstrained Feature Model (UFM) is a mathematical framework that enables closed-form approximations for minimal training loss and related performance measures in deep neural networks (DNNs). This paper leverages the UFM to provide qualitative insights into neural multivariate regression, a critical task in imitation learning, robotics, and reinforcement learning. Specifically, we address two key questions: (1) How do multi-task models compare to multiple single-task models in terms of training performance? (2) Can whitening and normalizing regression targets improve training performance? The UFM theory predicts that multi-task models achieve strictly smaller training MSE than multiple single-task models when the same or stronger regularization is applied to the latter, and our empirical results confirm these findings. Regarding whitening and normalizing regression targets, the UFM theory predicts that they reduce training MSE when the average variance across the target dimensions is less than one, and our empirical results once again confirm these findings. These findings highlight the UFM as a powerful framework for deriving actionable insights into DNN design and data pre-processing strategies.

Figures (10)

• Figure 1: Comparison of the training error of a single multi-task model with that of multiple single task models for different weight decay values after training with the standard parameter-regularized loss function.
• Figure 2: Comparison of the effect that target whitening and normalization have on training error for different weight decay values after training with the standard parameter-regularized loss function. The green curve (in short MSE) records the training error for different weight decay values after training with the original unprocessed targets.
• Figure 3: Effect of network architecture on multi‑ vs. single‑task training: Reacher. Comparison of the training error of a single multi-task model with that of multiple single task models for different weight decay values after training with the standard parameter-regularized loss function across different architectures. The architectures are denoted by by their layer sizes (input and output layers omitted for simplicity). The number of parameters increases from left to right and from top to bottom.
• Figure 4: Effect of network architecture on multi‑ vs. single‑task training: Swimmer. Comparison of the training error of a single multi-task model with that of multiple single task models for different weight decay values after training with the standard parameter-regularized loss function across different architectures. The architectures are denoted by by their layer sizes (input and output layers omitted for simplicity). The number of parameters increases from left to right and from top to bottom.
• Figure 5: Effect of network architecture on multi‑ vs. single‑task training: Hopper. Comparison of the training error of a single multi-task model with that of multiple single task models for different weight decay values after training with the standard parameter-regularized loss function across different architectures. The architectures are denoted by by their layer sizes (input and output layers omitted for simplicity). The number of parameters increases from left to right and from top to bottom.

Theorems & Definitions (25)

• Theorem 3.1
• Corollary 4.1
• Theorem 4.2
• Theorem 5.1
• Theorem 5.2
• Theorem 5.3
• Theorem 5.4
• proof : Proof of Theorem \ref{['gendim']}
• Corollary C.1
• proof