Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Matrix Calculus (for Machine Learning and Beyond)

Paige Bright, Alan Edelman, Steven G. Johnson

TL;DR

This work introduces matrix calculus as an extension of differential calculus to general vector spaces, framing derivatives as linear operators that map small input changes to first-order output changes. It develops the core toolkit of matrix derivatives, including the differential, gradient, Jacobian, directional derivatives, chain rule, and the differential product rule, with emphasis on computational efficiency through reverse-mode differentiation and automatic differentiation. The notes provide explicit formulas for matrix-valued functions (e.g., $A^{2}$, $A^{3}$, $A^{-1}$) and discuss representations via vectorization and Kronecker products to obtain Jacobians in a standard form. By contrasting forward- and reverse-mode AD and illustrating concrete matrix-function derivatives, the material equips learners with a practical, operator-centric framework for differentiating through large-scale matrix computations and simulations.

Abstract

This course, intended for undergraduates familiar with elementary calculus and linear algebra, introduces the extension of differential calculus to functions on more general vector spaces, such as functions that take as input a matrix and return a matrix inverse or factorization, derivatives of ODE solutions, and even stochastic derivatives of random functions. It emphasizes practical computational applications, such as large-scale optimization and machine learning, where derivatives must be re-imagined in order to be propagated through complicated calculations. The class also discusses efficiency concerns leading to "adjoint" or "reverse-mode" differentiation (a.k.a. "backpropagation"), and gives a gentle introduction to modern automatic differentiation (AD) techniques.

Matrix Calculus (for Machine Learning and Beyond)

TL;DR

This work introduces matrix calculus as an extension of differential calculus to general vector spaces, framing derivatives as linear operators that map small input changes to first-order output changes. It develops the core toolkit of matrix derivatives, including the differential, gradient, Jacobian, directional derivatives, chain rule, and the differential product rule, with emphasis on computational efficiency through reverse-mode differentiation and automatic differentiation. The notes provide explicit formulas for matrix-valued functions (e.g., , , ) and discuss representations via vectorization and Kronecker products to obtain Jacobians in a standard form. By contrasting forward- and reverse-mode AD and illustrating concrete matrix-function derivatives, the material equips learners with a practical, operator-centric framework for differentiating through large-scale matrix computations and simulations.

Abstract

This course, intended for undergraduates familiar with elementary calculus and linear algebra, introduces the extension of differential calculus to functions on more general vector spaces, such as functions that take as input a matrix and return a matrix inverse or factorization, derivatives of ODE solutions, and even stochastic derivatives of random functions. It emphasizes practical computational applications, such as large-scale optimization and machine learning, where derivatives must be re-imagined in order to be propagated through complicated calculations. The class also discusses efficiency concerns leading to "adjoint" or "reverse-mode" differentiation (a.k.a. "backpropagation"), and gives a gentle introduction to modern automatic differentiation (AD) techniques.
Paper Structure (16 sections, 1 theorem, 45 equations, 3 figures, 1 table)

This paper contains 16 sections, 1 theorem, 45 equations, 3 figures, 1 table.

Table of Contents

  1. Overview and Motivation
  2. Applications
  3. First Derivatives
  4. Intro: Matrix and Vector Product Rule
  5. Derivatives as Linear Operators
  6. Revisiting single-variable calculus
  7. Linear operators
  8. Directional derivatives
  9. Revisiting multivariable calculus, Part 1: Scalar-valued functions
  10. Revisiting multivariable calculus, Part 2: Vector-valued functions
  11. The Chain Rule
  12. Cost of Matrix Multiplication
  13. Beyond Multi-Variable Derivatives
  14. Jacobians of Matrix Functions
  15. Derivatives of matrix functions: Linear operators
  16. ...and 1 more sections

Key Result

Theorem 2

Let $A,B$ be two matrices. Then, we have the differential product rule for $AB$: By the differential of the matrix $A$, we think of it as a small (unconstrained) change in the matrix $A.$ Later, constraints may be places on the allowed perturbations.

Figures (3)

  • Figure 1: The essence of a derivative is linearization: predicting a small change $\delta f$ in the output $f(x)$ from a small change $\delta x$ in the input $x$, to first order in $\delta x$.
  • Figure 2: For a real-valued $f(x)$, the gradient $\nabla f$ is defined so that it corresponds to the "uphill" direction at a point $x$, which is perpendicular to the contours of $f$. Although this may not point exactly towards the nearest local maximum of $f$ (unless the contours are circular), "going uphill" is nevertheless the starting point for many computational-optimization algorithms to search for a maximum.
  • Figure 3: Matrix multiplication is associative---that is, $(AB)C = A(BC)$ for all $A,B,C$---but multiplying left-to-right can be much more efficient than right-to-left if the leftmost matrix has only one (or few) rows, as shown here. Correspondingly, the order in which you carry out the chain rule has dramatic consequences for the computational effort required. Left-to-right is known as "reverse mode" or "backpropagation", and is best suited to situations where there are many fewer outputs than inputs.

Theorems & Definitions (18)

  • Example 1
  • Theorem 2: Differential Product Rule
  • Remark 3
  • Example 4
  • Remark 5
  • Remark 6
  • Example 10
  • Example 11
  • Example 12
  • Example 13
  • ...and 8 more