Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pseudo-Invertible Neural Networks

Yamit Ehrlich, Nimrod Berman, Assaf Shocher

TL;DR

This work generalizes the Moore-Penrose pseudo-inverse to non-linear mappings by enforcing the first two Penrose identities and using bijective completion to select a unique non-linear PInv. It introduces Surjective PInv Neural Networks (SPNNs), a dimension-reducing architecture with a built-in PInv guaranteed by a two-phase training scheme and an auxiliary inverse network. The framework enables Non-Linear Back-Projection (NLBP), unifying linear null-space projection and gradient-based methods to perform zero-shot restoration of non-linear degradations, including semantic and attribute-level control in diffusion-generated images. By integrating SPNNs with a diffusion prior, the approach achieves stable, semantically constrained inversion without retraining the prior, with demonstrated capability on semantic reconstruction and attribute-controlled generation. The work provides a principled mathematical bridge for solving complex non-linear inverse problems in scientific computing and image synthesis.

Abstract

The Moore-Penrose Pseudo-inverse (PInv) serves as the fundamental solution for linear systems. In this paper, we propose a natural generalization of PInv to the nonlinear regime in general and to neural networks in particular. We introduce Surjective Pseudo-invertible Neural Networks (SPNN), a class of architectures explicitly designed to admit a tractable non-linear PInv. The proposed non-linear PInv and its implementation in SPNN satisfy fundamental geometric properties. One such property is null-space projection or "Back-Projection", $x' = x + A^\dagger(y-Ax)$, which moves a sample $x$ to its closest consistent state $x'$ satisfying $Ax=y$. We formalize Non-Linear Back-Projection (NLBP), a method that guarantees the same consistency constraint for non-linear mappings $f(x)=y$ via our defined PInv. We leverage SPNNs to expand the scope of zero-shot inverse problems. Diffusion-based null-space projection has revolutionized zero-shot solving for linear inverse problems by exploiting closed-form back-projection. We extend this method to non-linear degradations. Here, "degradation" is broadly generalized to include any non-linear loss of information, spanning from optical distortions to semantic abstractions like classification. This approach enables zero-shot inversion of complex degradations and allows precise semantic control over generative outputs without retraining the diffusion prior.

Pseudo-Invertible Neural Networks

TL;DR

This work generalizes the Moore-Penrose pseudo-inverse to non-linear mappings by enforcing the first two Penrose identities and using bijective completion to select a unique non-linear PInv. It introduces Surjective PInv Neural Networks (SPNNs), a dimension-reducing architecture with a built-in PInv guaranteed by a two-phase training scheme and an auxiliary inverse network. The framework enables Non-Linear Back-Projection (NLBP), unifying linear null-space projection and gradient-based methods to perform zero-shot restoration of non-linear degradations, including semantic and attribute-level control in diffusion-generated images. By integrating SPNNs with a diffusion prior, the approach achieves stable, semantically constrained inversion without retraining the prior, with demonstrated capability on semantic reconstruction and attribute-controlled generation. The work provides a principled mathematical bridge for solving complex non-linear inverse problems in scientific computing and image synthesis.

Abstract

The Moore-Penrose Pseudo-inverse (PInv) serves as the fundamental solution for linear systems. In this paper, we propose a natural generalization of PInv to the nonlinear regime in general and to neural networks in particular. We introduce Surjective Pseudo-invertible Neural Networks (SPNN), a class of architectures explicitly designed to admit a tractable non-linear PInv. The proposed non-linear PInv and its implementation in SPNN satisfy fundamental geometric properties. One such property is null-space projection or "Back-Projection", , which moves a sample to its closest consistent state satisfying . We formalize Non-Linear Back-Projection (NLBP), a method that guarantees the same consistency constraint for non-linear mappings via our defined PInv. We leverage SPNNs to expand the scope of zero-shot inverse problems. Diffusion-based null-space projection has revolutionized zero-shot solving for linear inverse problems by exploiting closed-form back-projection. We extend this method to non-linear degradations. Here, "degradation" is broadly generalized to include any non-linear loss of information, spanning from optical distortions to semantic abstractions like classification. This approach enables zero-shot inversion of complex degradations and allows precise semantic control over generative outputs without retraining the diffusion prior.
Paper Structure (46 sections, 2 theorems, 22 equations, 6 figures, 2 tables, 1 algorithm)

This paper contains 46 sections, 2 theorems, 22 equations, 6 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Deep Invertible Architectures.
  4. Surjective Architectures.
  5. Generalized Inversion of Non-Linear Operators.
  6. Zero-Shot Inverse Problems.
  7. Preliminaries
  8. Basic Notions of Invertibility
  9. The Linear Moore-Penrose Standard
  10. Back-Projection
  11. Generalized Non-Linear Inversion
  12. Iterative Back-Projection (IBP)
  13. Non-Linear Pseudo-Inverse
  14. Theoretical Justification
  15. Surjective Pseudo-invertible Neural Network (SPNN)
  16. ...and 31 more sections

Key Result

Theorem 5.1

The operator $g^\dagger$ defined in Eqs. eq:inverse_r-eq:inverse_affine is a strict right-inverse of $g$, satisfying $g(g^\dagger(y)) = y$ for all $y \in \mathbb{R}^d$.

Figures (6)

  • Figure 1: The SPNN Block Architecture.Top (Forward $g$): The input $x$ is rotated by $u$ and split. The null-space component $x_1$ modulates the signal $x_0$ via affine coupling layers ($s, t$) to produce the compressed output $y$. Bottom (Pseudo-Inverse $g^\dagger$): An auxiliary network $r$ predicts the missing null-space information $\hat{x}_1$ solely from $y$, allowing the reverse coupling flow to structurally reconstruct the high-dimensional pre-image $\hat{x}$.
  • Figure 2: Reconstruction from Semantics. We project a clean image (Left) into its 40-dimensional attribute vector (as a row-major grid, middle), see Appendix \ref{['app:attribs']} or fig. \ref{['fig:attrib_analysis']} for a list of attributes. Using only this low-dimensional semantic code, our NLBP guides the diffusion model to reconstruct a photorealistic face (Right) that faithfully preserves the attributes of the original subject.
  • Figure 3: Quantitative Attribute Reconstruction. Analysis of 100 test samples. Left: Binary agreement rate. Right: Mean absolute error in probability space.
  • Figure 4: Single Attribute Editing. By dynamically enforcing a specific target index in the attribute vector (e.g., forcing the "Eyeglasses" logit to be high), we can generate diverse samples that strictly adhere to the condition while hallucinating the rest of the image freely. This demonstrates that the SPNN has successfully disentangled the specific semantic attribute from the null-space.
  • Figure 5: Multi-Attribute Conditional Generation. simultaneously enforcing constraints by fixing multiple bits in the target $y$.
  • ...and 1 more figures

Theorems & Definitions (11)

  • Definition 4.1: Bijective Completion
  • Definition 4.2: The Natural Non-Linear PInv
  • Definition 4.3: Non-Linear Back-Projection
  • Claim 1
  • proof
  • Claim 2
  • proof
  • Theorem 5.1: Surjectivity / Exact Right-Inverse
  • proof
  • Theorem 5.2: Reflexive Consistency
  • ...and 1 more