Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalized Naive Bayes

Edith Alice Kovács, Anna Ország, Dániel Pfeifer, András Benczúr

TL;DR

The paper addresses NB's limitation from the conditional independence assumption by introducing Generalized Naive Bayes (GNB), a cherry-tree based extension that uses triplet clusters to better capture dependencies. It presents two learning algorithms, GNB-A (greedy) and GNB-O (optimal via maximum weighted arborescence), and proves that GNB offers a fit at least as good as NB in terms of KL divergence, with GNB-O achieving an optimal structure under a mild condition. The authors develop a classification process and integrated feature-selection methods, and validate their approach on several real medical datasets, frequently outperforming NB and TAN while maintaining interpretability through feature-importance scores. Overall, the work advances probabilistic graphical modeling for classification by providing transparent, information-theoretically grounded methods that improve accuracy and offer actionable feature insights in health-related tasks.

Abstract

In this paper we introduce the so-called Generalized Naive Bayes structure as an extension of the Naive Bayes structure. We give a new greedy algorithm that finds a good fitting Generalized Naive Bayes (GNB) probability distribution. We prove that this fits the data at least as well as the probability distribution determined by the classical Naive Bayes (NB). Then, under a not very restrictive condition, we give a second algorithm for which we can prove that it finds the optimal GNB probability distribution, i.e. best fitting structure in the sense of KL divergence. Both algorithms are constructed to maximize the information content and aim to minimize redundancy. Based on these algorithms, new methods for feature selection are introduced. We discuss the similarities and differences to other related algorithms in terms of structure, methodology, and complexity. Experimental results show, that the algorithms introduced outperform the related algorithms in many cases.

Generalized Naive Bayes

TL;DR

The paper addresses NB's limitation from the conditional independence assumption by introducing Generalized Naive Bayes (GNB), a cherry-tree based extension that uses triplet clusters to better capture dependencies. It presents two learning algorithms, GNB-A (greedy) and GNB-O (optimal via maximum weighted arborescence), and proves that GNB offers a fit at least as good as NB in terms of KL divergence, with GNB-O achieving an optimal structure under a mild condition. The authors develop a classification process and integrated feature-selection methods, and validate their approach on several real medical datasets, frequently outperforming NB and TAN while maintaining interpretability through feature-importance scores. Overall, the work advances probabilistic graphical modeling for classification by providing transparent, information-theoretically grounded methods that improve accuracy and offer actionable feature insights in health-related tasks.

Abstract

In this paper we introduce the so-called Generalized Naive Bayes structure as an extension of the Naive Bayes structure. We give a new greedy algorithm that finds a good fitting Generalized Naive Bayes (GNB) probability distribution. We prove that this fits the data at least as well as the probability distribution determined by the classical Naive Bayes (NB). Then, under a not very restrictive condition, we give a second algorithm for which we can prove that it finds the optimal GNB probability distribution, i.e. best fitting structure in the sense of KL divergence. Both algorithms are constructed to maximize the information content and aim to minimize redundancy. Based on these algorithms, new methods for feature selection are introduced. We discuss the similarities and differences to other related algorithms in terms of structure, methodology, and complexity. Experimental results show, that the algorithms introduced outperform the related algorithms in many cases.
Paper Structure (23 sections, 10 theorems, 60 equations, 11 figures, 5 tables, 2 algorithms)

This paper contains 23 sections, 10 theorems, 60 equations, 11 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Basic concepts
  4. Probabilistic graphical models used in the GNB structures
  5. Information theoretical concepts
  6. Generalized Naive Bayes
  7. Generalized Naive Bayes graph structure
  8. Greedy algorithm for finding a good fitting GNB structure
  9. Algorithm for finding the best fitting GNB structure
  10. On how the introduced GNB concepts are related to former improvements of NB
  11. The classification process and feature selection methods
  12. Feature selection, feature importance score
  13. Numerical Results
  14. Evaluation metrics
  15. Data preparation
  16. ...and 8 more sections

Key Result

Theorem 14

szantai2012hypergraphs The Kullback-Leibler divergence between the cherry tree approximation (eq:Cherry_tree) and the real distribution $P(\mathbf{X})$ is:

Figures (11)

  • Figure 1: An example construction of a third-order cherry tree.
  • Figure 2: Cherry tree structure represented as a chordal graph (first row), represented as junction tree (second row) and represented in a compact junction tree form (third row)
  • Figure 3: The maximum clique is of size four, which leads to a contradiction
  • Figure 4: Adding the vertex $0$ to each cluster
  • Figure 5: The initial construction of the cherry tree
  • ...and 6 more figures

Theorems & Definitions (52)

  • Definition 1
  • Remark 2
  • Definition 3
  • Remark 4
  • Remark 5
  • Definition 6
  • Remark 7
  • Definition 8
  • Remark 9
  • Definition 10
  • ...and 42 more