Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Coding for Intelligence from the Perspective of Category

Wenhan Yang, Zixuan Hu, Lilang Lin, Jiaying Liu, Ling-Yu Duan

TL;DR

This paper advances Coding for Intelligence (CfI) by casting coding and AI model learning within a category-theoretic framework grounded in three axioms and Minimal Description Length (MDL) as a compactness principle. It defines Ideal Coding and CfI via presheaves and the Yoneda perspective, linking intrinsic object relationships to downstream tasks through a task functor $F$ and a coding operator $C$. By instantiating $K$, proposing two modeling paradigms, and formulating an MDL-based objective $U$, the work bridges reconstruction, analytics, and multi-task/multi-modality coding, and discusses scalable pathways enabled by large foundation models. The authors review past work, propose two CfI pathways—Intelligent Bitstream Collaboration and Compressive Analytics—and provide preliminary results and directions, arguing that a category-theoretic view can unify diverse coding approaches and guide the design of intelligent, multi-task systems.

Abstract

Coding, which targets compressing and reconstructing data, and intelligence, often regarded at an abstract computational level as being centered around model learning and prediction, interweave recently to give birth to a series of significant progress. The recent trends demonstrate the potential homogeneity of these two fields, especially when deep-learning models aid these two categories for better probability modeling. For better understanding and describing from a unified perspective, inspired by the basic generally recognized principles in cognitive psychology, we formulate a novel problem of Coding for Intelligence from the category theory view. Based on the three axioms: existence of ideal coding, existence of practical coding, and compactness promoting generalization, we derive a general framework to understand existing methodologies, namely that, coding captures the intrinsic relationships of objects as much as possible, while ignoring information irrelevant to downstream tasks. This framework helps identify the challenges and essential elements in solving the specific derived Minimal Description Length (MDL) optimization problem from a broader range, providing opportunities to build a more intelligent system for handling multiple tasks/applications with coding ideas/tools. Centering on those elements, we systematically review recent processes of towards optimizing the MDL problem in more comprehensive ways from data, model, and task perspectives, and reveal their impacts on the potential CfI technical routes. After that, we also present new technique paths to fulfill CfI and provide potential solutions with preliminary experimental evidence. Last, further directions and remaining issues are discussed as well. The discussion shows our theory can reveal many phenomena and insights about large foundation models, which mutually corroborate with recent practices in feature learning.

Coding for Intelligence from the Perspective of Category

TL;DR

This paper advances Coding for Intelligence (CfI) by casting coding and AI model learning within a category-theoretic framework grounded in three axioms and Minimal Description Length (MDL) as a compactness principle. It defines Ideal Coding and CfI via presheaves and the Yoneda perspective, linking intrinsic object relationships to downstream tasks through a task functor and a coding operator . By instantiating , proposing two modeling paradigms, and formulating an MDL-based objective , the work bridges reconstruction, analytics, and multi-task/multi-modality coding, and discusses scalable pathways enabled by large foundation models. The authors review past work, propose two CfI pathways—Intelligent Bitstream Collaboration and Compressive Analytics—and provide preliminary results and directions, arguing that a category-theoretic view can unify diverse coding approaches and guide the design of intelligent, multi-task systems.

Abstract

Coding, which targets compressing and reconstructing data, and intelligence, often regarded at an abstract computational level as being centered around model learning and prediction, interweave recently to give birth to a series of significant progress. The recent trends demonstrate the potential homogeneity of these two fields, especially when deep-learning models aid these two categories for better probability modeling. For better understanding and describing from a unified perspective, inspired by the basic generally recognized principles in cognitive psychology, we formulate a novel problem of Coding for Intelligence from the category theory view. Based on the three axioms: existence of ideal coding, existence of practical coding, and compactness promoting generalization, we derive a general framework to understand existing methodologies, namely that, coding captures the intrinsic relationships of objects as much as possible, while ignoring information irrelevant to downstream tasks. This framework helps identify the challenges and essential elements in solving the specific derived Minimal Description Length (MDL) optimization problem from a broader range, providing opportunities to build a more intelligent system for handling multiple tasks/applications with coding ideas/tools. Centering on those elements, we systematically review recent processes of towards optimizing the MDL problem in more comprehensive ways from data, model, and task perspectives, and reveal their impacts on the potential CfI technical routes. After that, we also present new technique paths to fulfill CfI and provide potential solutions with preliminary experimental evidence. Last, further directions and remaining issues are discussed as well. The discussion shows our theory can reveal many phenomena and insights about large foundation models, which mutually corroborate with recent practices in feature learning.
Paper Structure (24 sections, 19 equations, 6 figures, 3 tables)

This paper contains 24 sections, 19 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Formulation
  3. Psychology Basis
  4. A Category Theory Perspective
  5. Definition
  6. Hint: How Coding Facilitates Intelligence
  7. Instances
  8. Instances of $K$
  9. Instances to Model $\rm{Hom}$ with the Guidance of $K$
  10. Specification: Minimal Depth Length Paradigm
  11. Discussion: from Relationship $K$ to Distance $L$
  12. Summary of Previous Works
  13. Diverse Coding Objects
  14. Evolving Models
  15. Multi-Task Modeling
  16. ...and 9 more sections

Figures (6)

  • Figure 1: The definition of (a) Ideal Coding: using $K$ to approximate $\rm{Hom}_\mathcal{W}$ for coding $X$' intrinsic relationship $H_\mathcal{W}(X)$; (b) Coding for Intelligence: $C(X)$ captures only $X$'s relationship that are only meaningful to $F$, under the constraint of MDL.
  • Figure 2: Specific instances under Coding for Intelligence.
  • Figure 3: Specified MDL formulation for different CfI instances. Image coding (A) and feature coding (B) aim to build a compact representation for given samples. Coding for machines (C) hopes to consider compactness from the perspective of the joint probability distribution. Coding for intelligence (D) expects to compress the whole thing from both perspectives of sample compactness and joint distribution compactness.
  • Figure 4: Deep learning has accelerated the development of AI and coding techniques, driving machine-driven image recognition beyond human cognitive capabilities and rapidly advancing end-to-end video encoding beyond traditional codecs.
  • Figure 5: (a) The recently emerging CfI instances that extend the categories in the dimensions of data, task, and model to form the joint optimization of multiple data/tasks via evolving models. (b) The recent technical trends that promote intelligence, which corresponds to the paradigms of modeling $\rm{Hom}$ under the guidance of $K$ in sequential conditional modeling or reparameterization.
  • ...and 1 more figures