From-scratch guide / Chapter 3 / Introductory

Gradient Descent from Scratch in Python

Gradient descent minimizes a differentiable objective by repeatedly moving the parameters opposite the gradient. In the example below, you will implement that update directly in NumPy and keep the full optimization path so you can inspect convergence.

How it works

w_(k+1) = w_k - alpha * grad g(w_k)

Here w_k is the current parameter, grad g(w_k) is the local slope of the objective, and alpha is the positive step size. The minus sign makes the update move downhill.

  1. 1Choose an initial parameter value and a positive step size.
  2. 2Evaluate the gradient at the current parameter.
  3. 3Subtract the step-size-scaled gradient.
  4. 4Repeat until the gradient or parameter change is sufficiently small.

Visual intuition

Normalized gradient descent steps moving across objective contours toward a minimum
The animation shows normalized gradient steps: direction comes from the negative gradient while every displayed step has a controlled length. The code example uses standard gradient descent, where gradient magnitude also affects the step.

NumPy implementation

This example is deterministic and uses NumPy for arrays and arithmetic, not a prebuilt optimizer or model implementation.

import numpy as np

def objective(w):
    return (w - 3.0) ** 2

def gradient(w):
    return 2.0 * (w - 3.0)

def gradient_descent(initial_w, step_size=0.2, iterations=30):
    w = float(initial_w)
    history = [w]

    for _ in range(iterations):
        w = w - step_size * gradient(w)
        history.append(w)

    return w, np.asarray(history)

minimum, path = gradient_descent(initial_w=8.0)

The objective has its minimum at w = 3, so the example has a result we can verify exactly.

The gradient function returns the local slope; the loop contains the complete learning rule rather than calling an optimizer.

Keeping history makes the optimization path available for plotting or convergence checks.

What can go wrong

A large step can overshoot

If alpha is too large, successive updates can bounce across the minimum or diverge.

A tiny step can stall progress

A very small alpha may be stable but require far more iterations than the same objective needs with a better-scaled step.

Geometry still matters

Flat, poorly conditioned, or non-convex objectives can make first-order progress slow or lead to different stationary points.

Continue with the book and source

This guide is grounded in Chapter 3, First-Order Optimization Techniques, and the official Machine Learning Refined notebooks. Use the chapter for context, then open the exact sources below for the full treatment and exercises.

Chapter 3: First-Order Optimization TechniquesRead the chapter overview and browse its complete resource set.Visual notebook workbenchCompare this animation with the other curated book notebooks.Gradient descent source notebookRead the Chapter 3 notebook that develops descent methods.Normalized descent companionOpen the notebook behind the normalized-step animation.Chapter 3 exercisesPractice first-order optimization with the official exercise notebook.Chapter PDF collectionOpen the official chapter PDF index for the book.Canonical GitHub repositoryBrowse the complete Machine Learning Refined source collection.