Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Machine Learning as Iterated Belief Change a la Darwiche and Pearl

Theofanis Aravanis

TL;DR

This work bridges iterated belief change and binary neural networks by reframing ANN training as a gradual sequence of belief-set transitions. It first critiques full-meet belief change and demonstrates that DP-compatible operators—lexicographic revision and moderate contraction—provide robust, iterated-update semantics aligned with training dynamics. By adopting Dalal's distance-based approach to beliefs, the authors show how intermediate belief states naturally emerge during learning and satisfy progressive distance properties. They validate the framework with binary-ANN experiments (including a logical function and MNIST) showing gradual convergence of symbolic knowledge, and prove DP-compatibility of learning under the SD condition, linking gradient-based optimization to principled belief revision.

Abstract

Artificial Neural Networks (ANNs) are powerful machine-learning models capable of capturing intricate non-linear relationships. They are widely used nowadays across numerous scientific and engineering domains, driving advancements in both research and real-world applications. In our recent work, we focused on the statics and dynamics of a particular subclass of ANNs, which we refer to as binary ANNs. A binary ANN is a feed-forward network in which both inputs and outputs are restricted to binary values, making it particularly suitable for a variety of practical use cases. Our previous study approached binary ANNs through the lens of belief-change theory, specifically the Alchourron, Gardenfors and Makinson (AGM) framework, yielding several key insights. Most notably, we demonstrated that the knowledge embodied in a binary ANN (expressed through its input-output behaviour) can be symbolically represented using a propositional logic language. Moreover, the process of modifying a belief set (through revision or contraction) was mapped onto a gradual transition through a series of intermediate belief sets. Analogously, the training of binary ANNs was conceptualized as a sequence of such belief-set transitions, which we showed can be formalized using full-meet AGM-style belief change. In the present article, we extend this line of investigation by addressing some critical limitations of our previous study. Specifically, we show that Dalal's method for belief change naturally induces a structured, gradual evolution of states of belief. More importantly, given the known shortcomings of full-meet belief change, we demonstrate that the training dynamics of binary ANNs can be more effectively modelled using robust AGM-style change operations -- namely, lexicographic revision and moderate contraction -- that align with the Darwiche-Pearl framework for iterated belief change.

Machine Learning as Iterated Belief Change a la Darwiche and Pearl

TL;DR

This work bridges iterated belief change and binary neural networks by reframing ANN training as a gradual sequence of belief-set transitions. It first critiques full-meet belief change and demonstrates that DP-compatible operators—lexicographic revision and moderate contraction—provide robust, iterated-update semantics aligned with training dynamics. By adopting Dalal's distance-based approach to beliefs, the authors show how intermediate belief states naturally emerge during learning and satisfy progressive distance properties. They validate the framework with binary-ANN experiments (including a logical function and MNIST) showing gradual convergence of symbolic knowledge, and prove DP-compatibility of learning under the SD condition, linking gradient-based optimization to principled belief revision.

Abstract

Artificial Neural Networks (ANNs) are powerful machine-learning models capable of capturing intricate non-linear relationships. They are widely used nowadays across numerous scientific and engineering domains, driving advancements in both research and real-world applications. In our recent work, we focused on the statics and dynamics of a particular subclass of ANNs, which we refer to as binary ANNs. A binary ANN is a feed-forward network in which both inputs and outputs are restricted to binary values, making it particularly suitable for a variety of practical use cases. Our previous study approached binary ANNs through the lens of belief-change theory, specifically the Alchourron, Gardenfors and Makinson (AGM) framework, yielding several key insights. Most notably, we demonstrated that the knowledge embodied in a binary ANN (expressed through its input-output behaviour) can be symbolically represented using a propositional logic language. Moreover, the process of modifying a belief set (through revision or contraction) was mapped onto a gradual transition through a series of intermediate belief sets. Analogously, the training of binary ANNs was conceptualized as a sequence of such belief-set transitions, which we showed can be formalized using full-meet AGM-style belief change. In the present article, we extend this line of investigation by addressing some critical limitations of our previous study. Specifically, we show that Dalal's method for belief change naturally induces a structured, gradual evolution of states of belief. More importantly, given the known shortcomings of full-meet belief change, we demonstrate that the training dynamics of binary ANNs can be more effectively modelled using robust AGM-style change operations -- namely, lexicographic revision and moderate contraction -- that align with the Darwiche-Pearl framework for iterated belief change.

Paper Structure

This paper contains 19 sections, 7 theorems, 1 equation, 9 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Formal Prerequisites
  3. The AGM Framework
  4. Axiomatic Characterization
  5. Semantic Characterization
  6. Full-Meet Belief Change
  7. Dalal's Revision Operator
  8. Belief Change as a Gradual Transition of Beliefs
  9. Distance between Belief Sets
  10. Gradual Transition of Beliefs via Dalal's Method
  11. Iterated Belief Change: The DP Approach
  12. Lexicographic Revision
  13. Moderate Contraction
  14. Binary Artificial Neural Networks
  15. Architecture
  16. ...and 4 more sections

Key Result

Theorem 4

A revision function $\ast$ satisfies postulates $(K\ast1)$--$(K\ast8)$ iff, for any theory $K$, there exists a total preorder $\preceq_K$ over $\mathbb{M}$, faithful to $K$, such that, for any sentence $\varphi\in\mathbb{L}$:

Figures (9)

  • Figure 1: A feed-forward ANN with a single hidden layer.
  • Figure 2: Graphical representation of lexicographic revision (a) and moderate contraction (b). A state of belief is modified (revised or contracted) by a sentence $\varphi$. $\varphi$-worlds are represented by green rectangles, while $\neg\varphi$-worlds are represented by red rectangles. The lower a rectangle is, the more plausible the represented worlds are relative to the state of belief --- thus, the lowest rectangles represent the worlds of the modified belief set.
  • Figure 3: A binary ANN with a single hidden layer. The network maps binary inputs $X_1,\dots,X_n$ to binary outputs $Y_1,\dots,Y_m$.
  • Figure 4: Representative MNIST images used in the experimental setup of Example \ref{['ex_mnist']}, along with their corresponding (binary) labels.
  • Figure 6: Schematic representation of the faithful preorders assigned by $\ast$ and $\mathbin{ \ooalign{ \hbox{.}\cr$-$} }$ at the belief sets $K_1$, $K_i\ast\varphi_{1}^{i}$, and $K_{i+1}$.
  • ...and 4 more figures

Theorems & Definitions (35)

  • Remark 1
  • Remark 2
  • Definition 3: Faithful Preorder, katsuno91
  • Theorem 4: katsuno91
  • Theorem 5: caridroit17
  • Definition 6: Full-Meet Belief Change
  • Remark 7
  • Definition 8: Difference between Worlds
  • Definition 9: Distance between Belief Sets and Worlds, dalal88
  • Definition 10: Dalal's Preorder, dalal88
  • ...and 25 more