Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Information Theoretic Perspective on Representation Learning

Deborah Pereg

TL;DR

The paper introduces an information-theoretic framework for last-layer representations in regression, defining representation rate, representation capacity, and rate-distortion, and derives bounds that unify noisy-input and compressed-output regimes. It shows that a bijective embedding ties the representation rate to the input entropy $H(X)$, while the presence of noise or output compression bounds the achievable rate by $I(X;Y)$ and the rate-distortion function $R(D)$, respectively. A channel- and source-coding–style correspondence (including a representation-rate–source-channel separation) yields a cohesive theory linking embedding capacity, data uncertainty, and distortion constraints. The results clarify fundamental limits on embedding cardinality and dimensionality and offer insight into the role of invertibility andmanifold structure in regression representations, with potential implications for the design of regression-focused encoders and large-model representations.

Abstract

An information-theoretic framework is introduced to analyze last-layer embedding, focusing on learned representations for regression tasks. We define representation-rate and derive limits on the reliability with which input-output information can be represented as is inherently determined by the input-source entropy. We further define representation capacity in a perturbed setting, and representation rate-distortion for a compressed output. We derive the achievable capacity, the achievable representation-rate, and their converse. Finally, we combine the results in a unified setting.

Information Theoretic Perspective on Representation Learning

TL;DR

The paper introduces an information-theoretic framework for last-layer representations in regression, defining representation rate, representation capacity, and rate-distortion, and derives bounds that unify noisy-input and compressed-output regimes. It shows that a bijective embedding ties the representation rate to the input entropy , while the presence of noise or output compression bounds the achievable rate by and the rate-distortion function , respectively. A channel- and source-coding–style correspondence (including a representation-rate–source-channel separation) yields a cohesive theory linking embedding capacity, data uncertainty, and distortion constraints. The results clarify fundamental limits on embedding cardinality and dimensionality and offer insight into the role of invertibility andmanifold structure in regression representations, with potential implications for the design of regression-focused encoders and large-model representations.

Abstract

An information-theoretic framework is introduced to analyze last-layer embedding, focusing on learned representations for regression tasks. We define representation-rate and derive limits on the reliability with which input-output information can be represented as is inherently determined by the input-source entropy. We further define representation capacity in a perturbed setting, and representation rate-distortion for a compressed output. We derive the achievable capacity, the achievable representation-rate, and their converse. Finally, we combine the results in a unified setting.
Paper Structure (36 sections, 8 theorems, 47 equations, 3 figures)

This paper contains 36 sections, 8 theorems, 47 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Remark 1.
  3. Capacity in neural networks.
  4. Background and Related Work
  5. Preliminaries
  6. Notations
  7. Classification vs Regression
  8. Confusion with compression and rate-distortion theory.
  9. Representation Capacity
  10. Channel Capacity
  11. Representation Capacity
  12. Bijective transformation
  13. The Embedding Space
  14. The Embedding Separation Distance
  15. Embedding Continuity
  16. ...and 21 more sections

Key Result

Theorem 1

Let $(u_n)_{n \in \mathbb{Z}}$ be a stationary ergodic process ranging over a finite alphabet $\mathcal{U}$, then there is a constant $H$, defining the entropy rate of the source,

Figures (3)

  • Figure 1: Illustration of the noise-free setting.
  • Figure 2: Illustration of a noisy environment problem setting.
  • Figure 3: Illustration of a rate-distortion setting.

Theorems & Definitions (14)

  • Theorem 1: Entropy and Ergodic Theory Breiman:1957
  • Theorem 2: Shannon McMillan (AEP)Breiman:1953
  • Proposition 3: Class-oriented representation-collapse for regression
  • Definition 4: Embedding Code
  • Definition 5: Embedding representation rate
  • Theorem 6: Embedding representation rate for a bijective mapping
  • Definition 7: Effective Support of the Embedding Space
  • Theorem 8: Embedding Capacity
  • Definition 9: Representation Capacity
  • Theorem 10: Generalization error for a noisy input
  • ...and 4 more