Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Penalizing Localized Dirichlet Energies in Low Rank Tensor Products

Paris A. Karakasis, Nicholas D. Sidiropoulos

TL;DR

The paper addresses generalization and regularization in regression/classification by analyzing low-rank tensor-product B-spline (TPBS) models and Dirichlet energy. It derives a closed-form expression for the Dirichlet energy of TPBS, reveals that global energy can be exponentially small even for perfect interpolation, and introduces a localized regularization LDE_\rho(g) to focus smoothing near data. It also develops two estimators for inference from incomplete observations using pretrained TPBS components and marginalization over unobserved features. Experiments on six datasets show TPBS with LDE is robust to overfitting and often outperforms neural networks in overfitting and missing-data regimes, highlighting TPBS as a strong, data-local regularization framework for high-dimensional regression and classification.

Abstract

We study low-rank tensor-product B-spline (TPBS) models for regression tasks and investigate Dirichlet energy as a measure of smoothness. We show that TPBS models admit a closed-form expression for the Dirichlet energy, and reveal scenarios where perfect interpolation is possible with exponentially small Dirichlet energy. This renders global Dirichlet energy-based regularization ineffective. To address this limitation, we propose a novel regularization strategy based on local Dirichlet energies defined on small hypercubes centered at the training points. Leveraging pretrained TPBS models, we also introduce two estimators for inference from incomplete samples. Comparative experiments with neural networks demonstrate that TPBS models outperform neural networks in the overfitting regime for most datasets, and maintain competitive performance otherwise. Overall, TPBS models exhibit greater robustness to overfitting and consistently benefit from regularization, while neural networks are more sensitive to overfitting and less effective in leveraging regularization.

Penalizing Localized Dirichlet Energies in Low Rank Tensor Products

TL;DR

The paper addresses generalization and regularization in regression/classification by analyzing low-rank tensor-product B-spline (TPBS) models and Dirichlet energy. It derives a closed-form expression for the Dirichlet energy of TPBS, reveals that global energy can be exponentially small even for perfect interpolation, and introduces a localized regularization LDE_\rho(g) to focus smoothing near data. It also develops two estimators for inference from incomplete observations using pretrained TPBS components and marginalization over unobserved features. Experiments on six datasets show TPBS with LDE is robust to overfitting and often outperforms neural networks in overfitting and missing-data regimes, highlighting TPBS as a strong, data-local regularization framework for high-dimensional regression and classification.

Abstract

We study low-rank tensor-product B-spline (TPBS) models for regression tasks and investigate Dirichlet energy as a measure of smoothness. We show that TPBS models admit a closed-form expression for the Dirichlet energy, and reveal scenarios where perfect interpolation is possible with exponentially small Dirichlet energy. This renders global Dirichlet energy-based regularization ineffective. To address this limitation, we propose a novel regularization strategy based on local Dirichlet energies defined on small hypercubes centered at the training points. Leveraging pretrained TPBS models, we also introduce two estimators for inference from incomplete samples. Comparative experiments with neural networks demonstrate that TPBS models outperform neural networks in the overfitting regime for most datasets, and maintain competitive performance otherwise. Overall, TPBS models exhibit greater robustness to overfitting and consistently benefit from regularization, while neural networks are more sensitive to overfitting and less effective in leveraging regularization.
Paper Structure (15 sections, 1 theorem, 15 equations, 1 figure, 14 tables)

This paper contains 15 sections, 1 theorem, 15 equations, 1 figure, 14 tables.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Low Rank Tensor Product models
  4. Dirichlet energy as a measure of complexity
  5. Main Contributions
  6. The Dirichlet energy of a low rank tensor model
  7. Proposed Regularization
  8. Handling Incomplete Observations
  9. Experimental Results
  10. Inference with full datasets
  11. Inference with incomplete datasets
  12. Conclusion
  13. Reproducing the Double Descent Curve Phenomenon
  14. Proof of Theorem 3.1
  15. Further results

Key Result

Theorem 3.1

For a function $\mathbf{g}$ represented by the low-rank tensor product model in (CPD), its Dirichlet energy can be expressed in closed form as where $\mathbf{s}(\mathbf{g}) \in \mathbb{R}^R_+$, $\mathbf{Z}_{r,s}(\mathbf{g}) \in \mathbb{R}^{R \times R}$, $\mathbf{A}^{(0)}(\mathbf{g}) \in \mathbb{R}^{R \times R}$, and $\mathbf{A}^{(1)}(\mathbf{g}) \in \mathbb{R}^{R \times R}$ are defined element-wi

Figures (1)

  • Figure 1: Double descent of test error with respect to network width for varying training set sizes on MNIST.

Theorems & Definitions (1)

  • Theorem 3.1