A notation that composes can be handed to a machine. Automatic differentiation is the chain rule carried out mechanically, one elementary step at a time, and reverse mode is that same process run from the output back to the inputs. When one number depends on many inputs, reverse mode is far cheaper, and that asymmetry is what backpropagation exploits. This chapter builds the engine, counts its cost, and runs it through the corpus's own generator.
Deisenroth, Faisal and Ong start §5.6 with a warning (p. 159). Write out the derivative of
by hand, and the expression for df/dx is much longer than f. Implemented naively, it can cost far more than the function it differentiates. Automatic differentiation avoids that. It is neither symbolic algebra nor a finite difference: it evaluates the exact derivative, up to machine precision, by attaching the chain rule to each elementary operation (p. 161). The authors credit backpropagation to Kelley (1960), Bryson (1961), Dreyfus (1962) and Rumelhart et al. (1986) CITED. The last of these arrived after this chapter was written and is read in Chapter 1.
Their Example 5.14 breaks f into six intermediate variables — a = x², b = exp(a), c = a + b, d = √c, e = cos c, f = d + e — and then walks back from f. Each step multiplies an incoming sensitivity by one local derivative and adds contributions where paths meet. That is the general rule (5.145), which the text names “the backpropagation of the gradient through the computation graph” (p. 164). Their observation that makes the method worth having is on p. 163: computed this way, the derivative is “of similar complexity as the computation of the function itself.”
The script for this chapter implements (5.143)–(5.145) directly. Each value records the values it came from and the local derivative along each edge, and a backward sweep in reverse topological order accumulates sensitivities. On Example 5.14 at x = 0.5 it returns −1.3604310786. That agrees with the book's closed form (5.110) to twelve decimal places and with a central difference to seven COMPUTED. Nothing in the engine knows calculus beyond the derivatives of +, ×, exp, cos and √. The chain rule does the rest.
There are two directions to run the chain. Forward mode carries a derivative alongside each value from the inputs out. It answers “how does everything change if this one input moves?”, so a full gradient over n inputs takes n passes. Reverse mode carries sensitivities from the output back. It answers “how does this one output depend on everything?”, so a full gradient takes one forward pass and one backward pass, whatever n is. The text states the consequence (p. 162): where inputs vastly outnumber outputs, as with the millions of parameters and one loss of a neural network, reverse mode “is computationally significantly cheaper than the forward mode.”
The script measures it on a function with one output and n inputs, counting graph operations in each mode:
| inputs n | forward mode | reverse mode | ratio |
|---|---|---|---|
| 10 | 380 | 83 | 4.6 |
| 100 | 39,800 | 893 | 44.6 |
| 1,000 | 3,998,000 | 8,993 | 444.6 |
Forward mode grows like n² here and reverse mode like n. Both give the same gradient to twelve places COMPUTED. With one input and many outputs the comparison reverses, and the honest statement is about shape: run the chain from whichever end is narrow. Backpropagation is reverse-mode automatic differentiation applied to a scalar loss, and that is all it is. Its power comes from how cheap one backward pass is, not from anything special about neural networks.
Chapter 2 differentiated G₃ = U₃ ∘ F₃ ∘ K₃ ∘ C₃ by multiplying Jacobians. The engine can do it without being told the Jacobians. Write the four operators as the Lean file defines them, run them on engine values, take the loss |G₃(s)|², and sweep back once:
| state s = (r, θ, z) | gradient by backprop | matches central differences |
|---|---|---|
| (1, 1, 0) | (8.7358, 6.0, 0.0) | yes |
| (2, 0.5, 0.3) | (11.2642, 25.0, 0.0) | yes |
The zero in the last place is a finding, not noise COMPUTED. C₃ overwrites z with r²θ, so nothing downstream depends on the input z, and its gradient is exactly zero at every state. A gradient method run on the corpus's generator would never move z. This is the same fact as Chapter 2's: the constraint erases the coordinate that kinetics shifts, which is why the C–K non-commutation vanishes under differentiation.
The second state lies above the fold, where F₃ reflects. The engine takes the branch the value selects and differentiates that branch. At r = κ exactly there is no derivative to take, and the engine would silently pick one side. MODEL — this is how every framework treats a corner, and it is why Chapter 2's warning about the fold matters in practice.