Iterative Minimisation by Steepest Descent

By the end you should be able to: Describe the steepest descent iteration, carry out its steps by hand, and explain the effect of the step size and why the method may find only a local minimum.

Setting $\nabla f = \mathbf 0$ and solving works when the equations are tractable. With a hundred variables and no closed form, it is hopeless.

Steepest descent gives up on solving exactly and walks downhill instead.

The idea

Module 3 established that $\nabla f$ points in the direction of steepest ascent. To go down, go the other way.

x_{n+1} = x_n - \alpha\, f'(x_n)

and for several variables

\mathbf{x}_{n+1} = \mathbf{x}_n - \alpha\, \nabla f(\mathbf{x}_n)

The number $\alpha > 0$ is the step size — the learning rate in machine learning. Start somewhere, repeat, stop when the steps become negligible.

A worked run

Minimise $f(x)=x^2$ , whose answer we already know is $x=0$ . Take $f'(x)=2x$ , $\alpha = 0.1$ , starting at $x_0 = 1$ :

$n$	$x_n$	$f'(x_n)=2x_n$	$x_{n+1} = x_n - 0.1(2x_n)$
0	1	2	0.8
1	0.8	1.6	0.64
2	0.64	1.28	0.512
3	0.512	1.024	0.4096

Each step multiplies $x$ by $0.8$ , so $x_n = 0.8^n$ . It approaches zero without ever arriving — which is normal. Iterative methods converge; they do not terminate.

The step size decides everything

$\alpha$	Behaviour on $f=x^2$
Too small (0.001)	Correct direction, painfully slow
Well chosen (0.1)	Steady geometric convergence
$\alpha = 0.5$	$x_{n+1}=0$ in one step — exact, by luck
Too large (1.5)	$x_{n+1}=-2x_n$ : overshoots, oscillates, diverges

With $\alpha = 1.5$ starting from $x_0 = 1$ : $1 \to -2 \to 4 \to -8$ . The method runs away from the minimum it is meant to find. Too large a step is not merely slow — it is wrong.

Stopping

There is no exact arrival, so choose a criterion:

$|\nabla f|$ below a tolerance — the ground is flat enough
$|\mathbf x_{n+1}-\mathbf x_n|$ below a tolerance — progress has stalled
a maximum iteration count, so a failing run still ends

The honest limitation

Steepest descent goes downhill from where it starts. Drop it into a valley and it finds the bottom of that valley. If a deeper one exists elsewhere, it will never learn of it — nothing in the method looks beyond the local slope.

This is the practical face of the local-versus-global distinction from Module 3, and it is why training a neural network from two different random initialisations can produce two different models.

Why it dominates in practice

Each step needs only the gradient — no second derivatives, no matrix inversion, no solving of systems. For a function of a million variables that is the difference between feasible and not. Nearly all modern machine learning is a refinement of this loop.