Neural Networks and Deep Learning

Lecture 2: Training: backpropagation and optimisation

Lecture 2

Training: backpropagation and optimisation

Prof. dr. Daniel Traian PELE

Bucharest University of Economic Studies (ASE), Department of Statistics and Econometrics
Romanian Academy, Institute for Economic Forecasting (IPE)

A neural network is only as good as its training. This lecture opens the black box of `loss.backward()` and `optimizer.step()`: how the gradient of a loss with thousands or millions of weights is computed exactly by backpropagation, and how gradient descent (GD) and its stochastic, momentum and adaptive variants turn that gradient into good weights. Every claim is measured on real data: 20,640 Californian census block groups for regression, 28,924 card payments for fraud detection and the daily S&P 500 for a 1-day value at risk (VaR).

What you will be able to do

  • Write gradient descent as the solution method of empirical risk minimisation and explain why conditioning sets its speed.
  • Derive backpropagation for a multilayer perceptron (MLP), compute it by hand and check it against automatic differentiation.
  • Explain vanishing gradients and choose an initialisation that keeps the signal alive.
  • Compare stochastic gradient descent (SGD), momentum, root mean square propagation (RMSProp), Adam and learning-rate schedules with several seeds.
  • Train and backtest a quantile network for the 99 % VaR of the S&P 500.

Lecture plan

  1. From the loss to the gradientTraining is the minimisation of an average loss over millions of parameters; gradient descent solves it one small step at a time, and the shape of the loss surface decides how fast.
  2. BackpropagationBackpropagation is the chain rule applied from the loss backwards; it gives every partial derivative for about the cost of one more forward pass.
  3. Stochastic optimisationMini-batches, the learning rate, momentum, adaptive steps and schedules decide how fast and how reliably the network learns; we measure each on California housing and on card fraud, with several seeds.
  4. Case study: value at risk with a quantile networkThe same network, trained with another loss, becomes a risk model: with the pinball loss an MLP forecasts the 1 % quantile of the next-day S&P 500 return, and we backtest it from 2020.

From the loss to the gradient

Training is the minimisation of an average loss over millions of parameters; gradient descent solves it one small step at a time, and the shape of the loss surface decides how fast.

Training a network is a high-dimensional, non-convex minimisation

Slide 5 in the PDF

There is no formula for the weights of a network. We start somewhere and repeat \(\boldsymbol\theta_{t+1}=\boldsymbol\theta_t-\eta\nabla\mathcal L(\boldsymbol\theta_t)\): to first order, \(\mathcal L(\boldsymbol\theta-\eta\mathbf g)=\mathcal L(\boldsymbol\theta)-\eta\|\mathbf g\|^2\), so a small step against the gradient lowers the loss. Our MLP 64-64 has 4,801 parameters for 12,384 training block groups.

The data of this lecture: houses, card payments and the S&P 500

Slide 6 in the PDF

California housing is split at random 60/20/20 into 12,384, 4,128 and 4,128 block groups. The card-fraud data of the Université libre de Bruxelles (ULB) and Worldline, published on the Open Machine Learning platform (OpenML), keep all 492 frauds and a 10 % sample of legitimate payments; their inputs are principal component analysis (PCA) scores, and weights restore the true fraud rate of 0.173 %. Fraud and S&P 500 are split in time order.

Badly scaled inputs turn the loss into a narrow valley

Slide 7 in the PDF

The mean squared error (MSE) of the one-variable regression \(\hat y=a+b\cdot\text{income}\) is a real quadratic surface. On raw income its Hessian has condition number 104.3: the contours are long ellipses, and after 60 steps plain GD is still at MSE 1.486, momentum at 0.704. On standardised income the condition number is 1 and GD reaches the minimum, 0.701, almost at once.

Chart: Badly scaled inputs turn the loss into a narrow valley

Gradient descent and momentum on the real loss surface, raw and standardised income.

Source: StatLib, California housing, 1990 census (Pace and Barry, 1997). Data: 20,640 block groups, 1990 census.

In practice: standardise every input on the training set before training any network.

On a quadratic, the step size must stay below 2/lambda_max

Slide 8 in the PDF

Near a minimum, \(\mathcal L\approx\mathcal L^*+\tfrac12(\boldsymbol\theta-\boldsymbol\theta^*)^\top\mathbf H(\boldsymbol\theta-\boldsymbol\theta^*)\) and each eigen-direction of the Hessian contracts by \(1-\eta\lambda_k\) per step. Convergence needs \(0<\eta<2/\lambda_{\max}\), and with the best fixed step the error shrinks by \((\kappa-1)/(\kappa+1)\), slow when the condition number \(\kappa\) is large.

Section summary

Gradient descent moves against the gradient; the learning rate must respect the steepest curvature, and poorly scaled inputs make the flat directions painfully slow.

Backpropagation

Backpropagation is the chain rule applied from the loss backwards; it gives every partial derivative for about the cost of one more forward pass. We compute it by hand, check it numerically and see why deep sigmoid networks stop learning.

Backpropagation is the chain rule, applied from the loss backwards

Slide 10 in the PDF

With \(\mathbf z^{(l)}=\mathbf W^{(l)}\mathbf h^{(l-1)}+\mathbf b^{(l)}\) and \(\mathbf h^{(l)}=a(\mathbf z^{(l)})\), the error signal travels backwards as

\[\boldsymbol\delta^{(l)}=\big(\mathbf W^{(l+1)\top}\boldsymbol\delta^{(l+1)}\big)\odot a'(\mathbf z^{(l)}),\qquad \frac{\partial\ell}{\partial\mathbf W^{(l)}}=\boldsymbol\delta^{(l)}\mathbf h^{(l-1)\top}\]
(2.1)

Each weight gradient in (2.1) is the error of its layer times the layer's input, the same form as the logistic gradient of Lecture 1. The forward pass must keep every \(\mathbf z^{(l)}\) and \(\mathbf h^{(l)}\), which is why memory limits the batch size on a graphics processing unit (GPU).

Backpropagation by hand on one real block group

Slide 11 in the PDF

For one real test block group with standardised income −0.21 and latitude −0.67, a net with two rectified linear units (ReLU) computes \(z_1=\) 0.433 and \(z_2=\) −0.093, outputs \(\hat y=\) −0.377 against \(y=\) 2.067, and has output error \(\delta^{(2)}=\) −2.444. Unit 2 is off (\(z_2<0\)), so it passes no gradient and its weights do not move; PyTorch autograd gives the same numbers.

Reverse mode costs about one extra forward pass, whatever the number of weights

Slide 12 in the PDF

One backward sweep delivers all \(P\) partial derivatives of a scalar loss; finite differences would need \(P\) extra passes. A gradient check on the MLP 64-64, with central differences in double precision on 30 random weights, finds a largest relative error of \(7.0\times10^{-7}\).

Vanishing gradients in deep sigmoid networks

Slide 13 in the PDF

The backward signal is a product of Jacobians \(\mathbf W^\top\operatorname{diag}(a')\). In a 10-layer net on one real batch, the first-layer gradient norm is \(8.5\times10^{-7}\) with sigmoids against \(7.5\times10^{0}\) in the last layer; with ReLU and He initialisation the first layer still gets \(3.1\times10^{0}\).

Chart: Vanishing gradients in deep sigmoid networks

Gradient norm per layer at initialisation, 10 hidden layers, mean of 5 seeds.

Source: StatLib, California housing, 1990 census (Pace and Barry, 1997). Data: 10 hidden layers of 64 units, one batch of 512 training block groups, 5 seeds.

Initialisation that keeps the variance of the signal constant across layers

Slide 14 in the PDF

If \(\operatorname{Var}(z^{(l)})=n_{l-1}\operatorname{Var}(W)\,\mathbb E[(h^{(l-1)})^2]\) is to stay constant, Xavier initialisation sets \(\operatorname{Var}(W)=2/(n_{l-1}+n_l)\) for tanh and sigmoid, and He initialisation \(\operatorname{Var}(W)=2/n_{l-1}\) for ReLU. Weights drawn from \(N(0,0.01^2)\) give a first-layer gradient of only \(6.9\times10^{-13}\): the network does not learn.

Section summary

Backpropagation gives exact gradients cheaply; whether they reach the first layers depends on the activation and on the scale of the initial weights.

Stochastic optimisation

Mini-batches, the learning rate, momentum, adaptive steps and schedules decide how fast and how reliably the network learns; we measure each on California housing and on card fraud, with several seeds.

A mini-batch gives an unbiased, noisy gradient at a fraction of the cost

Slide 16 in the PDF

The mini-batch gradient \(\mathbf g_t=\frac1B\sum_{i\in\mathcal B_t}\nabla\ell_i\) is unbiased for the full gradient and its variance falls roughly as \(1/B\). With step sizes satisfying \(\sum\eta_t=\infty\) and \(\sum\eta_t^2<\infty\), stochastic gradient descent converges to a stationary point (Robbins and Monro, 1951).

Effect of the learning rate on training

Slide 17 in the PDF

With plain SGD, 30 epochs and batches of 128, the validation MSE is 1.093 at \(\eta=10^{-4}\), worse than linear regression (0.500), and 0.281 at \(\eta=\) 0.1; from \(\eta=\) 0.3 at least one seed diverges.

Chart: Effect of the learning rate on training

Validation MSE after 30 epochs of SGD for nine learning rates and three seeds.

Source: StatLib, California housing, 1990 census (Pace and Barry, 1997). Data: 12,384 training and 4,128 validation block groups; 30 epochs.

In practice: tune the learning rate first, on a log grid.

Effect of batch size: progress per epoch and per second

Slide 18 in the PDF

After 5 epochs of Adam the validation MSE is 0.294 with batches of 16, 0.367 with 128 and 0.870 with 1,024: small batches make 774 steps per epoch, large ones 13. Per second the picture reverses: 40 epochs take 25.8, 3.7 and 1.0 seconds.

Chart: Effect of batch size: progress per epoch and per second

Validation MSE by epoch (left) and by gradient step (right) for three batch sizes.

Source: StatLib, California housing, 1990 census (Pace and Barry, 1997). Data: Adam, learning rate 0.001, 40 epochs.

Momentum: a running average of past gradients

Slide 19 in the PDF

Momentum keeps a velocity \(\mathbf v_{t+1}=\beta\mathbf v_t+\mathbf g_t\) and steps along it. Zig-zag components cancel, the consistent direction accumulates, and on a quadratic the best momentum improves the contraction from \((\kappa-1)/(\kappa+1)\) to \((\sqrt\kappa-1)/(\sqrt\kappa+1)\) (Polyak, 1964).

Adaptive methods give each weight its own step size

Slide 20 in the PDF

AdaGrad divides each weight's step by the root of the sum of its squared gradients, which only grows; root mean square propagation (RMSProp) replaces the sum by an exponential average, \(s_t=\rho s_{t-1}+(1-\rho)g_t^2\), so the step adapts to the current region.

Adam: momentum, RMSProp and a bias correction

Slide 21 in the PDF

Adam keeps exponential averages of the gradient and of its square, corrects them for starting at zero and steps by their ratio:

\[\hat m_t=\frac{m_t}{1-\beta_1^t},\quad \hat v_t=\frac{v_t}{1-\beta_2^t},\quad \theta_{t+1}=\theta_t-\eta\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}\]
(2.2)

With the defaults \(\beta_1=0.9\), \(\beta_2=0.999\), the step (2.2) is roughly bounded by \(\eta\) per weight. Adam with decoupled weight decay (AdamW) shrinks the weights directly instead of adding the penalty to the gradient (Loshchilov and Hutter, 2019).

On California housing, tuned optimisers reach similar errors and adaptive methods converge more reliably

Slide 22 in the PDF

Each optimiser gets the best of three learning rates on validation, then 5 new seeds. After 10 epochs the validation MSE is 0.342 for SGD, 0.310 with momentum, 0.313 for RMSProp, 0.305 for Adam and 0.304 for AdamW; linear regression stays at 0.500.

Chart: On California housing, tuned optimisers reach similar errors and adaptive methods converge more reliably

Validation MSE per epoch of five optimisers at their tuned learning rates, mean of 5 seeds.

Source: StatLib, California housing, 1990 census (Pace and Barry, 1997). Data: 12,384 training and 4,128 validation block groups.

Across seeds, Adam and AdamW are far more stable than plain SGD

Slide 23 in the PDF

The test MSE over 5 seeds is 0.314 (standard deviation 0.091) for SGD, 0.272 (0.010) for Adam and 0.266 (0.006) for AdamW. The paired t-test of SGD against Adam gives \(t=\) 1.01 and \(p=\) 0.371: the mean gain is not established with 5 seeds, the stability gain is plain to see.

Chart: Across seeds, Adam and AdamW are far more stable than plain SGD

Test MSE of five seeds per optimiser; bars are means; dashed: linear regression.

Source: StatLib, California housing, 1990 census (Pace and Barry, 1997). Data: 4,128 test block groups.

Learning-rate schedules: step, cosine and one-cycle

Slide 24 in the PDF

Step decay multiplies the rate by \(\gamma\) every \(s\) steps, cosine annealing follows \(\eta_t=\tfrac{\eta_0}{2}(1+\cos\pi t/T)\) (Loshchilov and Hutter, 2017), and the one-cycle schedule warms up linearly and then decays (Smith and Topin, 2019). Warm-up protects the first steps, when the optimiser's statistics are still unreliable.

Decaying the learning rate lowers the final error

Slide 25 in the PDF

With SGD and momentum at a peak rate of 0.01, the final validation MSE is 0.279 with a constant rate, 0.270 with step decay, 0.266 with cosine annealing and 0.266 with one-cycle; on the test set 0.286 against 0.265 for constant and cosine.

Chart: Decaying the learning rate lowers the final error

Learning-rate schedules (left) and validation MSE (right), mean of 3 seeds.

Source: StatLib, California housing, 1990 census (Pace and Barry, 1997). Data: SGD with momentum 0.9, peak rate 0.01, 3 seeds.

On card fraud, the optimiser determines the speed of learning

Slide 26 in the PDF

With only 360 frauds in the first 60 % of the two days, the area under the precision-recall curve (PR-AUC) on the validation hours after one epoch is 0.004 for plain SGD, 0.488 with momentum and 0.592 with Adam; on the test hours the three reach 0.786, 0.823 and 0.803, against 0.789 for logistic regression.

Chart: On card fraud, the optimiser determines the speed of learning

Validation PR-AUC per epoch on card fraud, three optimisers, mean and range of 5 seeds.

Source: ULB Machine Learning Group and Worldline, credit-card transactions (OpenML 1597). Data: September 2013, 2 days; train = first 33.4 hours, validation to hour 40.3; 5 seeds.

Section summary

Tune the learning rate first; momentum and adaptive methods converge faster and more reliably; a decaying schedule lowers the final error; and every comparison needs several seeds.

Case study: value at risk with a quantile network

The same network, trained with another loss, becomes a risk model: with the pinball loss an MLP forecasts the 1 % quantile of the next-day S&P 500 return, and we backtest it from 2020.

The pinball loss turns any network into a quantile forecaster

Slide 29 in the PDF

The VaR is minus the conditional \(\tau\)-quantile of the next-day return, and the pinball loss \(\rho_\tau(u)=u(\tau-\mathbb 1\{u<0\})\) is minimised by that quantile (Koenker and Bassett, 1978). Our MLP 32-32 uses the last return, its absolute value, the 5-day mean and the 5-, 22- and 66-day volatility; it is trained from 8 April 2005 to 2016, stopped on 2017-2019 and averaged over 5 seeds. The benchmarks are 250-day historical simulation and an exponentially weighted moving average (EWMA) volatility with a normal quantile.

Breaches of the quantile-MLP VaR during and after the 2020 pandemic are close to the nominal rate

Slide 30 in the PDF

Over 1,716 test days, from 3 January 2020 to 30 October 2026, a correct 99 % VaR should be breached about 17.2 times. The quantile MLP is breached 18 times, historical simulation 31 and EWMA 41; the MLP's VaR widens within days when volatility jumps, as in March 2020.

Chart: Breaches of the quantile-MLP VaR during and after the 2020 pandemic are close to the nominal rate

S&P 500 daily log returns in the test period with the 99 % VaR of the quantile MLP and of historical simulation; red: MLP violations.

Source: Yahoo Finance, daily quotes. Data: 3 Jan 2020 to 30 Oct 2026, trading days (test period).

The Kupiec test accepts the MLP and rejects both benchmarks

Slide 31 in the PDF

Under a correct VaR the violations are Bernoulli with probability \(\tau\); the Kupiec likelihood-ratio statistic gives \(p=\) 0.840 for the MLP, 0.003 for historical simulation and 0.000 for EWMA. The pinball losses, the proper score for quantiles, are 0.0420, 0.0577 and 0.0473.

Training hardware: a laptop GPU is ten times faster than its main processor on a large MLP

Slide 32 in the PDF

On this course's laptop one Adam step of an MLP 2048-2048 takes 333.6 ms on the central processing unit (CPU) and 32.5 ms on its GPU through Metal Performance Shaders (MPS), 10.3 times faster. The small tabular nets of Lectures 1 to 3 are faster on one CPU thread: copying tiny batches costs more than it saves.

Explore it visually

Free interactive tools and illustrated guides for this lecture; most run in the browser.

  • TensorFlow Playground: press the step button to see one gradient update at a time, then compare learning rates 0.03, 3 and 0.0001: the loss settles, oscillates, or barely moves.
  • 3Blue1Brown, chapters 2–4: gradient descent, the intuition of backpropagation and its calculus, with animations of the chain rule.

Section summary

The loss defines the product: with the pinball loss the MLP gives a 99 % VaR that passes the Kupiec test while both standard benchmarks fail it on the same days.

Videos: the intuition behind the mathematics (in English)

Abbreviations

AdamWAdam with decoupled weight decay
CPUcentral processing unit
ESexpected shortfall
EWMAexponentially weighted moving average
GDgradient descent
GPUgraphics processing unit
LLMlarge language model
MLPmultilayer perceptron
MPSMetal Performance Shaders (Apple GPU back-end of PyTorch)
MSEmean squared error
OpenMLOpen Machine Learning platform
PCAprincipal component analysis
PR-AUCarea under the precision-recall curve
ReLUrectified linear unit
RMSProproot mean square propagation
SGDstochastic gradient descent
ULBUniversité libre de Bruxelles
VaRvalue at risk

References

  • Baydin, A. G., Pearlmutter, B. A., Radul, A. A. & Siskind, J. M. (2018). Automatic differentiation in machine learning: a survey. Journal of Machine Learning Research, 18(153), 1–43. arXiv:1502.05767
  • Bengio, Y., Simard, P. & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157–166. doi:10.1109/72.279181
  • Bishop, C. M. & Bishop, H. (2024). Deep Learning: Foundations and Concepts. Springer. doi:10.1007/978-3-031-45468-4
  • Dal Pozzolo, A., Caelen, O., Johnson, R. A. & Bontempi, G. (2015). Calibrating probability with undersampling for unbalanced classification. IEEE Symposium Series on Computational Intelligence, 159–166. doi:10.1109/SSCI.2015.33
  • Duchi, J., Hazan, E. & Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12, 2121–2159.
  • Glorot, X. & Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. Proceedings of AISTATS, PMLR 9, 249–256.
  • He, K., Zhang, X., Ren, S. & Sun, J. (2015). Delving deep into rectifiers. IEEE International Conference on Computer Vision, 1026–1034. doi:10.1109/ICCV.2015.123
  • Jheng, S.-L., Tak, R., Găman, Ș., Mazurencu-Marinescu-Pele, M. & Pele, D. T. (2026). Can foundation models manage risk? Zero-shot VaR and ES forecasting with conformal calibration in CEE markets. Economic Computation and Economic Cybernetics Studies and Research, 60(2), 267–290. doi:10.24818/18423264/60.2.26.14
  • Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M. & Tang, P. T. P. (2017). On large-batch training for deep learning: Generalization gap and sharp minima. International Conference on Learning Representations. arXiv:1609.04836
  • Kingma, D. P. & Ba, J. (2015). Adam: A method for stochastic optimization. International Conference on Learning Representations. arXiv:1412.6980
  • Koenker, R. & Bassett, G. (1978). Regression quantiles. Econometrica, 46(1), 33–50. doi:10.2307/1913643
  • Kupiec, P. H. (1995). Techniques for verifying the accuracy of risk measurement models. The Journal of Derivatives, 3(2), 73–84. doi:10.3905/jod.1995.407942
  • Loshchilov, I. & Hutter, F. (2017). SGDR: Stochastic gradient descent with warm restarts. International Conference on Learning Representations. arXiv:1608.03983
  • Loshchilov, I. & Hutter, F. (2019). Decoupled weight decay regularization. International Conference on Learning Representations. arXiv:1711.05101
  • Pace, R. K. & Barry, R. (1997). Sparse spatial autoregressions. Statistics & Probability Letters, 33(3), 291–297. doi:10.1016/S0167-7152(96)00140-X
  • Paszke, A. et al. (2019). PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32. arXiv:1912.01703
  • Pele, D. T., Bolovăneanu, V., Lin, M.-B., Ren, R., Ginavar, A. T., Spilak, B., Andrei, A.-V., Toma, F.-M., Lessmann, S. & Härdle, W. K. (2026). In the beginning was the Word: LLM-VaR and LLM-ES. Expert Systems with Applications, 295, 128676. doi:10.1016/j.eswa.2025.128676
  • Polyak, B. T. (1964). Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 4(5), 1–17. doi:10.1016/0041-5553(64)90137-5
  • Robbins, H. & Monro, S. (1951). A stochastic approximation method. The Annals of Mathematical Statistics, 22(3), 400–407. doi:10.1214/aoms/1177729586
  • Rumelhart, D. E., Hinton, G. E. & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536. doi:10.1038/323533a0
  • Smith, L. N. & Topin, N. (2019). Super-convergence: Very fast training of neural networks using large learning rates. Proceedings of SPIE, 11006. arXiv:1708.07120
  • Tieleman, T. & Hinton, G. (2012). Lecture 6.5 — RMSProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning.

Text written for the web version of the lecture; numbers, charts and sources come from the slides and their Quantlets.