Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Maximum Entropy Linear Feature Inversion

Paul M Baggenstoss

TL;DR

The paper addresses the problem of recovering $\mathbf{x}$ from a dimension-reduced linear projection $\mathbf{z}=\mathbf{W}^T\mathbf{x}$ by maximizing entropy. It develops a unified MaxEnt framework with two complementary formulations: an idealized distribution on the feasible set $\mathcal{M}(\mathbf{z})$ and a tractable asymptotic surrogate whose mean $\bar{\mathbf{x}}_z=\lambda(\mathbf{W}\mathbf{h}_z)$ matches the ideal solution for large $N$. The approach supports multiple data ranges ($\mathbb{R}^N$, $\mathbb{P}^N$, $\mathbb{U}^N$) and various priors within a single exponential family, enabling cases like $[0,1]$ data via the truncated exponential (TED). Empirically, it recovers classical results in unbounded and positive-valued settings and yields improved reconstructions for doubly-bounded data, including an auto-encoder style demonstration that preserves the content of handwritten-like images.

Abstract

We revisit the classical problem of inverting dimension-reducing linear mappings using the maximum entropy (MaxEnt) criterion. In the literature, solutions are problem-dependent, inconsistent, and use different entropy measures. We propose a new unified approach that not only specializes to the existing approaches, but offers solutions to new cases, such as when data values are constrained to [0, 1], which has new applications in machine learning.

On Maximum Entropy Linear Feature Inversion

TL;DR

The paper addresses the problem of recovering from a dimension-reduced linear projection by maximizing entropy. It develops a unified MaxEnt framework with two complementary formulations: an idealized distribution on the feasible set and a tractable asymptotic surrogate whose mean matches the ideal solution for large . The approach supports multiple data ranges (, , ) and various priors within a single exponential family, enabling cases like data via the truncated exponential (TED). Empirically, it recovers classical results in unbounded and positive-valued settings and yields improved reconstructions for doubly-bounded data, including an auto-encoder style demonstration that preserves the content of handwritten-like images.

Abstract

We revisit the classical problem of inverting dimension-reducing linear mappings using the maximum entropy (MaxEnt) criterion. In the literature, solutions are problem-dependent, inconsistent, and use different entropy measures. We propose a new unified approach that not only specializes to the existing approaches, but offers solutions to new cases, such as when data values are constrained to [0, 1], which has new applications in machine learning.
Paper Structure (18 sections, 18 equations, 1 figure, 2 tables)

This paper contains 18 sections, 18 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Problem Statement
  3. Maximum Entropy Approaches
  4. Idealized Approach
  5. Main Idea
  6. Mathematical Details
  7. Data Ranges and Prior Distributions
  8. Unified Exponential form of Prior distribution
  9. Asymptotic Approach
  10. Main Idea
  11. Surrogate Distribution
  12. Examples
  13. Unbounded Data
  14. Positive-Valued Data
  15. Doubly-Bounded Data
  16. ...and 3 more sections

Figures (1)

  • Figure 1: $28\times 28$ images reconstructed from $7\times 7$ DCT features. Columns, from left to right: (a) Original, (b) reconstruction using inverse DCT, (c) MaxEnt reconstruction using positive/Exponential assumption, (d) MaxEnt reconstruction using doubly-bounded/Uniform assumption.