Gradient descent explained simply: how machines actually learn

Almost every machine learning model learns with gradient descent. What it is, how the learning rate and batches work, why optimisers such as Adam help, and what goes wrong, explained without heavy maths.

Gradient descent is the method most machine learning models use to learn. The model measures how wrong it is with a loss function, calculates which direction to change each parameter to reduce that loss, the gradient, and takes a small step in that direction. Repeating this thousands of times gradually finds parameters that make good predictions.

From a simple price-prediction model to the largest language models, the same idea does the learning. Understanding it helps you debug training, choose settings sensibly and follow what is happening when a model refuses to improve.

What is the intuition behind gradient descent?

Imagine standing on a hillside in thick fog, trying to reach the valley floor. You cannot see the bottom, but you can feel which way the ground slopes under your feet. So you take a small step downhill, feel again, and repeat. The height of the hill is the loss, your position is the model’s parameters, and the slope under your feet is the gradient.

What is a loss function?

A loss function turns “how wrong is the model” into a single number. For predicting prices, a common loss is mean squared error: the average squared difference between predicted and actual prices. For classification, cross-entropy loss measures how much probability the model gave to the correct answer. Lower loss means better predictions on the training data.

What is the learning rate?

The learning rate is the size of each step. Too large, and the model overshoots the valley and bounces around or diverges. Too small, and training crawls or gets stuck. Choosing it well is one of the most important practical decisions in training, and schedules that start larger and shrink over time are common.

What are batch, stochastic and mini-batch gradient descent?

Why do optimisers such as Adam help?

Plain gradient descent uses the same step size for every parameter. Optimisers such as momentum remember previous steps to roll through small bumps, and Adam adapts the step size for each parameter based on the history of its gradients. Adam and its variants are the default choice for training most neural networks, including language models.

How does this connect to backpropagation?

In a neural network with millions of parameters, you need the gradient for every one. Backpropagation computes them efficiently using the chain rule, working backwards from the loss. Gradient descent then uses those gradients to update the weights. Frameworks such as PyTorch do the calculus automatically. Our guide to neural networks for beginners explains the full training loop.

What goes wrong during training?

What does gradient descent look like in code?

w = 0.0                       # one parameter, starting guess
learning_rate = 0.01
for step in range(1000):
    prediction = w * x
    loss = ((prediction - y) ** 2).mean()
    gradient = (2 * (prediction - y) * x).mean()
    w = w - learning_rate * gradient   # step downhill

Real models do the same thing with millions or billions of parameters at once, which is why GPUs and good frameworks matter.

How do you choose a good learning rate in practice?

Start with a commonly used default for your optimiser, run a short training job, and watch the loss. If it explodes, lower the rate; if it barely moves, raise it. A learning rate range test, where you increase the rate gradually over a short run and see where loss falls fastest, is a quick way to find a sensible starting point. Warm-up, starting small for the first steps, helps large models train stably.

Does gradient descent find the best possible model?

Not necessarily the single best one, but usually a good one. For simple models such as linear regression, the loss surface has one bowl-shaped minimum and gradient descent finds it. For neural networks, the surface is complicated, but in practice the many good solutions found by gradient descent generalise well, especially with sensible regularisation and enough data.

How much maths do you need?

Derivatives, partial derivatives and the chain rule, plus comfort with vectors and matrices. You do not need to derive every formula by hand, but you should understand what a gradient means. Our guide on how much maths machine learning needs covers this by role. Program Zero teaches gradients and optimisation in its maths for machine learning track, then applies them in deep learning, including backpropagation from scratch.

Program Zero runs live online, so you can learn this from anywhere in India: metros such as Kolkata, Bengaluru, Mumbai, Delhi NCR and Hyderabad, growing cities such as Pune, Ahmedabad, Jaipur and Lucknow, and smaller towns such as Asansol, Durgapur, Siliguri and Guwahati. Every learner joins the same evening classes, at the same fee.

Frequently asked questions

Related reading