Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Finite Markov chains and Monte-Carlo Methods: An Undergraduate Introduction

Soumik Pal, Tim Mesikepp

Abstract

This is a free textbook suitable for a one-semester course on Markov chains, covering basics of finite-state chains, many classical models, asymptotic behavior and mixing times, Monte Carlo methods, and martingales and harmonic functions. It is designed to fill a gap in the literature by being suitable for undergraduates; much of the theory is thus built from the ground up, with only basic probability and linear algebra assumed. We take as our basic framework the first four chapters of the classic Levin-Peres text "Markov Chains and Mixing Times," generously expanding to make an exposition suitable for an undergraduate audience. We also incorporate over a hundred exercises and problems, along with a rich set of accompanying illustrations. Suggested homework sets are found in an appendix. Updated editions will periodically appear as new versions of this submission.

Finite Markov chains and Monte-Carlo Methods: An Undergraduate Introduction

Abstract

This is a free textbook suitable for a one-semester course on Markov chains, covering basics of finite-state chains, many classical models, asymptotic behavior and mixing times, Monte Carlo methods, and martingales and harmonic functions. It is designed to fill a gap in the literature by being suitable for undergraduates; much of the theory is thus built from the ground up, with only basic probability and linear algebra assumed. We take as our basic framework the first four chapters of the classic Levin-Peres text "Markov Chains and Mixing Times," generously expanding to make an exposition suitable for an undergraduate audience. We also incorporate over a hundred exercises and problems, along with a rich set of accompanying illustrations. Suggested homework sets are found in an appendix. Updated editions will periodically appear as new versions of this submission.
Paper Structure (78 sections, 47 theorems, 587 equations, 43 figures)

This paper contains 78 sections, 47 theorems, 587 equations, 43 figures.

Table of Contents

  1. Markov Chains
  2. Random walks on graphs
  3. What are graphs?
  4. Simple symmetric random walks on graphs
  5. Weighted walks on graphs
  6. Transition matrices
  7. Markov chains
  8. A first look at asymptotic behavior
  9. Irreducibility and aperiodicity
  10. Stationary distributions $\pi$
  11. $\pi$ for simple symmetric random walks on graphs
  12. Hitting times and return times
  13. Expected hitting times
  14. Expected return times
  15. Time-reversibility
  16. ...and 63 more sections

Key Result

Theorem 1.1

For any $k \in \mathbb{N} = \{1,2,3, \ldots\}$ and $i,j \in \Omega$, the $(i,j)$-entry of the $k$th power of the transition matrix $P$.

Figures (43)

  • Figure 1: A graph with five vertices
  • Figure 2: The 6-cycle graph. In general, the $n$-cycle has $n$ vertices, $n$ edges, and each vertex is connected to two neighbors, forming one "loop" around the entire graph.
  • Figure 3: Another embedding of the 6-cycle. As a graph, this is identical to the hexagonal embedding of the 6-cycle in Figure \ref{['Fig:Hexagon']}.
  • Figure 4: The integers $\mathbb{Z}$ as an infinite graph.
  • Figure 5: An Erdős-Rényi random graph with $n=50$ vertices and edge inclusion probability $p=0.2$.
  • ...and 38 more figures

Theorems & Definitions (143)

  • Definition 1.1
  • Definition 1.2
  • Definition 1.3
  • Example 1.1
  • Theorem 1.1
  • proof
  • Example 1.2: The 6-cycle, revisited
  • Definition 1.4
  • Example 1.3
  • Theorem 1.2
  • ...and 133 more