Sharpness-aware Second-order Latent Factor Model for High-dimensional and Incomplete Data

TL;DR

This work tackles learning from high-dimensional and incomplete interaction data using latent-factor models that are hard to optimize due to bilinear nonconvexity. It introduces SSLF, a sharpness-aware second-order latent factor model that leverages Hessian-vector products via a Gauss-Newton approximation and a hidden mapping trick to enable scalable, second-order optimization while promoting flat minima. Empirical results on Yelp and MovieLens 1M show SSLF outperforms SGD- and Adam-based baselines in RMSE and requires far fewer optimization epochs, indicating better generalization and efficiency. The approach offers a practical pathway to robust, scalable representation learning in HDI settings with sparse observations.

Abstract

Second-order Latent Factor (SLF) model, a class of low-rank representation learning methods, has proven effective at extracting node-to-node interaction patterns from High-dimensional and Incomplete (HDI) data. However, its optimization is notoriously difficult due to its bilinear and non-convex nature. Sharpness-aware Minimization (SAM) has recently proposed to find flat local minima when minimizing non-convex objectives, thereby improving the generalization of representation-learning models. To address this challenge, we propose a Sharpness-aware SLF (SSLF) model. SSLF embodies two key ideas: (1) acquiring second-order information via Hessian-vector products; and (2) injecting a sharpness term into the curvature (Hessian) through the designed Hessian-vector products. Experiments on multiple industrial datasets demonstrate that the proposed model consistently outperforms state-of-the-art baselines.