Paper Review 8: SGDR: SGD with warm restarts(+ theory behind the Grad descent algorithm)

🤔 What you need to know before reading paper?

This section is essential if you want a quick refresher or need to learn the basics of deep learning.

“If I have seen further, it is by standing on the shoulders of giants.” — Sir Isaac Newton

Isaac Newton famously said he only saw further by standing on the shoulders of giants. Before we look at where the authors are taking us, we need to take a quick look at the “giants” they are standing on—the basic background of this field.

Lets start..Gradient descent?

Training of DNN(deep neural networks) is a problem of minimizing a function f : Rⁿ → R

🤔 f : Rⁿ → R means: you feed f a vector of n real numbers, and it spits out one real number.

🌐 What is Function?

🌐 Vector

🌐 Real number

Examples:

1. n = 2: f : R² → R takes a pair (x, y) and returns one number.

This is like a surface — for every point (x, y) on a plane, you get a height. Think of a hill: input is your (latitude, longitude)-style location, output is the elevation there.

“To optimize f we iteratively adjust xₜ ∈ ℝⁿ (the parameter vector at time step t) using gradient information ∇fₜ(xₜ).”

This process is almost always written as:

• xₜ₊₁ — the new parameter vector (next step)
• xₜ — current parameter vector
• η (eta) — the learning rate / step size (how big a step to take)
• ∇fₜ(xₜ) — the gradient at the current point
• The minus sign means: move against the gradient, i.e., downhill

🤔 But, Where is the data points from train set lives?

f (or fₜ) is not just a function of x — it’s secretly a function of both the parameters and the data. The full picture is usually:

• z_i = (input_i, label_i) — a single data point (e.g., an image and its class)
• Bₜ — the batch of data points sampled at time step t (could be one example, a mini-batch, or the whole dataset)
• ℓ(x; z_i) — the loss on a single example, given parameters x
• fₜ(x) — the average loss over the batch, at parameters x

So fₜ(xₜ) = “plug the current parameters xₜ into the model, run them against batch Bₜ, and compute the average loss.”

Also good to know that gradient of a loss function is a function itself

There is also second order methods, instead of just using GD

Second-order methods use curvature to correct the step:

🤔 what is H?

The Hessian is the matrix of second derivatives of f — it tells you about the curvature of the loss surface, not just its slope.

From Gradient to Hessian

The Hessian is the matrix of second derivatives of f — it tells you about the curvature of the loss surface, not just its slope.

• Gradient ∇f(x) — a vector of first derivatives (∂f/∂x₁, ∂f/∂x₂, …, ∂f/∂xₙ). Tells you the slope in each direction — which way is downhill.

• Hessian H — a matrix of second derivatives. Tells you how the slope itself is changing — i.e., whether the surface is curving like a bowl, a saddle, a ridge, etc.

🤔 Why It’s a Problem in Practice

For a model with n parameters:

Gradient ∇f - n numbers - cheap

Hessian H - n × n numbers - expensive to compute

Inverse Hessian H⁻¹ - n × n - very expensive to compute (roughly O(n³))

🤔 you will probably ask a question, why we say that Hessian is a maxtrix

if f(x) = x³ (single variable, n = 1):

• f'(x) = 3x² (gradient — 1 number, since n=1)
• f''(x) = 6x (Hessian — which is just… a number)

ANSWER

So when n = 1, the “matrix” has exactly one entry, and it collapses down to what you already know as the ordinary second derivative. Nothing new happens — this is the degenerate case.

🤔 Where It Actually Becomes a Matrix

The Hessian only becomes a real matrix (more than 1 entry) when you have more than one variable — because now you need to ask not just “how does the slope change,” but “how does the slope in each direction change as I move in each direction” — and there are multiple combinations of that.

SGD with momentum

Instead of stepping purely based on the current gradient, you keep a “memory” of past gradients that keeps pushing you in a consistent direction.

The Two Equations

update the velocity:

update the parameters using velocity:

Compare this to plain gradient descent, which you already know:

The difference: plain GD moves you directly by the (scaled) gradient. Momentum instead updates a separate velocity vector v, and that is what moves you.