Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On efficiently computable functions, deep networks and sparse compositionality

Tomaso Poggio

TL;DR

The paper links efficient Turing computability with compositional sparsity by proving that any $f:[0,1]^d\to\mathbb{R}^m$ computable in time polynomial in the input precision induces a bounded-fan-in DAG representation and a correspondingly efficient neural approximant at fixed precision. It provides a precise discretization framework via $Q_n$ and $F_{n,m_{\mathrm{out}}}$, shows how a TM-to-circuit unrolling yields poly$(n+m_{\mathrm{out}})$-size networks, and demonstrates neural emulation of gates with error control to achieve $2^{-m_{\mathrm{out}}}$ accuracy. The work connects to compositional approximation theory and depth-efficiency results, illustrating that composition through low-arity, sparse structures enables favorable approximation rates and hierarchical optimization. It also bridges discrete Boolean circuits to smooth neural representations through a smooth lift, and discusses autoregressive universality as an associated learning-theoretic perspective. Overall, the results illuminate how efficient computation implies sparsity that underpins both expressive efficiency and optimization over sparse, hierarchical representations in neural approximants at finite precision.

Abstract

We show that \emph{efficient Turing computability} at any fixed input/output precision implies the existence of \emph{compositionally sparse} (bounded-fan-in, polynomial-size) DAG representations and of corresponding neural approximants achieving the target precision. Concretely: if $f:[0,1]^d\to\R^m$ is computable in time polynomial in the bit-depths, then for every pair of precisions $(n,m_{\mathrm{out}})$ there exists a bounded-fan-in Boolean circuit of size and depth $\poly(n+m_{\mathrm{out}})$ computing the discretized map; replacing each gate by a constant-size neural emulator yields a deep network of size/depth $\poly(n+m_{\mathrm{out}})$ that achieves accuracy $\varepsilon=2^{-m_{\mathrm{out}}}$. We also relate these constructions to compositional approximation rates \cite{MhaskarPoggio2016b,poggio_deep_shallow_2017,Poggio2017,Poggio2023HowDS} and to optimization viewed as hierarchical search over sparse structures.

On efficiently computable functions, deep networks and sparse compositionality

TL;DR

The paper links efficient Turing computability with compositional sparsity by proving that any computable in time polynomial in the input precision induces a bounded-fan-in DAG representation and a correspondingly efficient neural approximant at fixed precision. It provides a precise discretization framework via and , shows how a TM-to-circuit unrolling yields poly-size networks, and demonstrates neural emulation of gates with error control to achieve accuracy. The work connects to compositional approximation theory and depth-efficiency results, illustrating that composition through low-arity, sparse structures enables favorable approximation rates and hierarchical optimization. It also bridges discrete Boolean circuits to smooth neural representations through a smooth lift, and discusses autoregressive universality as an associated learning-theoretic perspective. Overall, the results illuminate how efficient computation implies sparsity that underpins both expressive efficiency and optimization over sparse, hierarchical representations in neural approximants at finite precision.

Abstract

We show that \emph{efficient Turing computability} at any fixed input/output precision implies the existence of \emph{compositionally sparse} (bounded-fan-in, polynomial-size) DAG representations and of corresponding neural approximants achieving the target precision. Concretely: if is computable in time polynomial in the bit-depths, then for every pair of precisions there exists a bounded-fan-in Boolean circuit of size and depth computing the discretized map; replacing each gate by a constant-size neural emulator yields a deep network of size/depth that achieves accuracy . We also relate these constructions to compositional approximation rates \cite{MhaskarPoggio2016b,poggio_deep_shallow_2017,Poggio2017,Poggio2023HowDS} and to optimization viewed as hierarchical search over sparse structures.

Paper Structure

This paper contains 13 sections, 10 theorems, 12 equations.

Table of Contents

  1. Introduction
  2. A note on uniformity
  3. Setup: Precision, Discretization, and Families
  4. From Polytime to Bounded-Fan-In Circuits
  5. Neural Emulation at Fixed Precision
  6. Relation to Compositional Approximation
  7. Depth-efficiency phenomena.
  8. Example.
  9. Interpretation.
  10. Relation to Autoregressive Universality
  11. Boolean vs. Real: From Discrete Circuits to Smooth Networks
  12. Discussion.
  13. Acknowledgments.

Key Result

Lemma 3.1

If $f$ is efficiently computable (Def. def:ETC), then for each $(n,m_{\mathrm{out}})$ there exists a bounded-fan-in Boolean circuit $C_{n,m_{\mathrm{out}}}$ of size and depth $\mathrm{poly}(n+m_{\mathrm{out}})$ computing $F_{n,m_{\mathrm{out}}}$.

Theorems & Definitions (18)

  • Definition 2.1: Efficient Turing computability at precision
  • Definition 2.2: Compositionally sparse family
  • Definition 2.3: Sparse polynomial (operational)
  • Lemma 3.1: Bounded-fan-in circuits for $F_{n,m_{\mathrm{out}}}$
  • Lemma 3.2: Circuits are compositional DAGs
  • Remark 3.3: Uniformity, briefly
  • Lemma 4.1: Gate emulation
  • Lemma 4.2: Circuit $\Rightarrow$ neural network
  • Theorem 4.3: Efficient computability $\Rightarrow$ compositional sparsity and neural approximation
  • Remark 4.4: Upgrading to uniform approximation
  • ...and 8 more