Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Joint Bayesian Parameter and Model Order Estimation for Low-Rank Probability Mass Tensors

Joseph K. Chege, Arie Yeredor, Martin Haardt

TL;DR

A novel Bayesian framework for estimating the joint PMF and automatically inferring its rank from observed data is presented, specifying a Bayesian PMF estimation model and employing appropriate prior distributions for the model parameters, allowing for tuning-free rank inference via a single training run.

Abstract

Obtaining a reliable estimate of the joint probability mass function (PMF) of a set of random variables from observed data is a significant objective in statistical signal processing and machine learning. Modelling the joint PMF as a tensor that admits a low-rank canonical polyadic decomposition (CPD) has enabled the development of efficient PMF estimation algorithms. However, these algorithms require the rank (model order) of the tensor to be specified beforehand. In real-world applications, the true rank is unknown. Therefore, an appropriate rank is usually selected from a candidate set either by observing validation errors or by computing various likelihood-based information criteria, a procedure which is computationally expensive for large datasets. This paper presents a novel Bayesian framework for estimating the joint PMF and automatically inferring its rank from observed data. We specify a Bayesian PMF estimation model and employ appropriate prior distributions for the model parameters, allowing for tuning-free rank inference via a single training run. We then derive a deterministic solution based on variational inference (VI) to approximate the posterior distributions of various model parameters. Additionally, we develop a scalable version of the VI-based approach by leveraging stochastic variational inference (SVI) to arrive at an efficient algorithm whose complexity scales sublinearly with the size of the dataset. Numerical experiments involving both synthetic data and real movie recommendation data illustrate the advantages of our VI and SVI-based methods in terms of estimation accuracy, automatic rank detection, and computational efficiency.

Joint Bayesian Parameter and Model Order Estimation for Low-Rank Probability Mass Tensors

TL;DR

A novel Bayesian framework for estimating the joint PMF and automatically inferring its rank from observed data is presented, specifying a Bayesian PMF estimation model and employing appropriate prior distributions for the model parameters, allowing for tuning-free rank inference via a single training run.

Abstract

Obtaining a reliable estimate of the joint probability mass function (PMF) of a set of random variables from observed data is a significant objective in statistical signal processing and machine learning. Modelling the joint PMF as a tensor that admits a low-rank canonical polyadic decomposition (CPD) has enabled the development of efficient PMF estimation algorithms. However, these algorithms require the rank (model order) of the tensor to be specified beforehand. In real-world applications, the true rank is unknown. Therefore, an appropriate rank is usually selected from a candidate set either by observing validation errors or by computing various likelihood-based information criteria, a procedure which is computationally expensive for large datasets. This paper presents a novel Bayesian framework for estimating the joint PMF and automatically inferring its rank from observed data. We specify a Bayesian PMF estimation model and employ appropriate prior distributions for the model parameters, allowing for tuning-free rank inference via a single training run. We then derive a deterministic solution based on variational inference (VI) to approximate the posterior distributions of various model parameters. Additionally, we develop a scalable version of the VI-based approach by leveraging stochastic variational inference (SVI) to arrive at an efficient algorithm whose complexity scales sublinearly with the size of the dataset. Numerical experiments involving both synthetic data and real movie recommendation data illustrate the advantages of our VI and SVI-based methods in terms of estimation accuracy, automatic rank detection, and computational efficiency.
Paper Structure (19 sections, 47 equations, 8 figures, 2 tables)

This paper contains 19 sections, 47 equations, 8 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Notation
  4. Problem Statement
  5. Variational Bayesian PMF Estimation
  6. Model Specification
  7. Posterior Approximation using Variational Inference
  8. Posterior distribution $q^{\ast}_{\lambda}(\boldsymbol{\lambda})$ of the loading vector $\boldsymbol{\lambda}$
  9. Posterior distribution $q^{\ast}_{n,r}(\bm{a}_{n,r})$ of the factor matrix columns $\bm{a}_{n,r}$
  10. Posterior distribution $q^{\ast}_z(\bm{z}_t)$ of the latent variable $\bm{z}_t$
  11. Computing the ELBO
  12. Algorithm: VB-PMF
  13. Synthetic-Data Experiments
  14. Performance with varying number of observations
  15. Performance with varying ranks and outage probabilities
  16. ...and 4 more sections

Figures (8)

  • Figure 1: Probabilistic graphical model representation of the joint distribution $p(\bm{Y},\bm{\Theta})$. The local and global random variables are represented by the larger circles, whereas the hyperparameters are depicted by the smaller solid dots. The shaded circle denotes that $y_t$ is an observed realization. The plate represents $T$ instances, of which only {$y_t$, $z_t$} are shown explicitly.
  • Figure 2: Typical convergence behavior of the VB-PMF algorithm, initialized with $R=23$ components, for a rank-5 PMF tensor. Left: convergence of the evidence lower bound (ELBO) (left y-axis) and evolution of the rank (right y-axis). Right: elements of the estimated loading vector $\widehat{\boldsymbol{\lambda}}$ after convergence.
  • Figure 3: Joint PMF estimation performance of VB-PMF, compared to other PMF estimation approaches, versus the number of observations $T$, averaged over 50 trials.
  • Figure 4: Rank estimation performance versus the outage probability $p$.
  • Figure 5: Mean KLD between the estimated and true PMF tensors versus the outage probability $p$.
  • ...and 3 more figures