Time Series Analysis

Chapter 9: Machine learning for time series

Chapter 9: Machine learning for time series

Daniel Traian PELE

Bucharest University of Economic Studies
IDA Institute Digital Assets
Blockchain Research Center
AI4EFin Artificial Intelligence for Energy Finance
Romanian Academy, Institute for Economic Forecasting
MSCA Digital Finance

Acronyms used in this material (1/2)

Slide 1 in the PDF
  • AI: Artificial Intelligence
  • AR: AutoRegressive (model)
  • ARIMA: AutoRegressive Integrated Moving Average (model)
  • AUC: Area Under the (ROC) Curve
  • BET: Bucharest Exchange Trading (index)
  • CPU: Central Processing Unit
  • DHR: Dynamic Harmonic Regression
  • DM: Diebold–Mariano (test)
  • ENTSO-E: European Network of Transmission System Operators for Electricity
  • ES-RNN: Exponential Smoothing – Recurrent Neural Network (hybrid model)
  • ETS: Error, Trend, Seasonality (exponential smoothing state space models)
  • EU: European Union
  • GB: Gradient Boosting (ensemble of regression trees)
  • GBRT: Gradient Boosted Regression Trees
  • GDP: Gross Domestic Product
  • GLM: Generalised Linear Model
  • HAR: Heterogeneous AutoRegressive (model)
  • HICP: Harmonised Index of Consumer Prices
  • LSTM: Long Short-Term Memory (network)
  • MAE: Mean Absolute Error
  • MASE: Mean Absolute Scaled Error
  • MDI: Mean Decrease Impurity

Acronyms used in this material (2/2)

Slide 2 in the PDF
  • MIMO: Multiple-Input Multiple-Output (forecasting strategy)
  • ML: Machine Learning
  • MLP: MultiLayer Perceptron
  • MSE: Mean Squared Error
  • OLS: Ordinary Least Squares
  • OLS-3: OLS with three predictors: size, book-to-market and momentum (Gu, Kelly and Xiu, 2020)
  • OWA: Overall Weighted Average (of the relative sMAPE and MASE, M4 competition)
  • PCR: Principal Component Regression
  • PLS: Partial Least Squares
  • QLIKE: Quasi-LIKElihood (loss function)
  • RF: Random Forest
  • RMSE: Root Mean Squared Error
  • RMSSE: Root Mean Squared Scaled Error
  • RNN: Recurrent Neural Network
  • ROC: Receiver Operating Characteristic (curve)
  • RW: Random Walk (forecast)
  • SARIMA: Seasonal ARIMA (model)
  • SES: Simple Exponential Smoothing
  • SGD: Stochastic Gradient Descent
  • US: United States
  • WRMSSE: Weighted Root Mean Squared Scaled Error (M5 competition)

Today's question and route

Slide 3 in the PDF

The chapter asks whether flexible learning algorithms, such as random forests, gradient boosting and neural networks, forecast a time series more accurately than ARIMA, ETS and the naive forecast, and how such a claim is checked. Machine learning here means a family of algorithms that learn a prediction rule from examples. For a time series the examples are not given: they have to be built from the past of the series itself, and this construction is the first topic of the chapter.

The route has three stages. The first turns forecasting into supervised learning: lag features, rolling windows, calendar features and the three strategies for multi-step forecasts. The second presents the methods and the way to validate them: walk-forward validation and leakage, ridge and lasso, trees and their ensembles, small neural networks and the LSTM. The third covers prediction intervals, the difference between local and global models, four applications and the M4 and M5 competitions. The chapter builds on the accuracy measures and benchmarks of Chapter 0 and on the time-series cross-validation and the Diebold–Mariano test of Chapter 4.

In brief

  • Question: can random forests, gradient boosting and neural networks forecast a time series better than ARIMA, ETS and the naive forecast, and how do we check it honestly?
    • machine learning (ML): algorithms that learn a prediction rule from examples; for time series, the examples are built from the past of the series
  • Route of the chapter
    • forecasting as supervised learning: lags, rolling windows, calendar features; recursive, direct and multi-output strategies
    • walk-forward validation and leakage; ridge and lasso; trees, random forest, gradient boosting; small neural networks and the LSTM
    • prediction intervals; local and global models; four applications and the M4, M5 competitions
  • We build on Chapter 0 (accuracy measures, benchmarks) and Chapter 4 (time-series cross-validation, Diebold–Mariano)

Learning outcomes

Slide 4 in the PDF

The first outcomes concern the data: the reader should be able to turn a series into a supervised-learning table that uses only information available at the forecast origin, and to produce forecasts several steps ahead with the recursive, direct and multi-output strategies. The third outcome is methodological: to validate a model walk-forward and to recognise leakage and look-ahead bias, the two errors that make ML forecasts look better than they are.

The remaining outcomes concern the methods and their evaluation. The reader should be able to explain ridge, lasso, regression trees, random forests, gradient boosting, the multilayer perceptron and the LSTM, together with the parameter that controls the complexity of each; to build prediction intervals by quantile regression and conformal prediction; and to compare ML forecasts with statistical benchmarks on the same test period, using the MASE and the Diebold–Mariano test.

In brief

  • Turn a time series into a supervised-learning table without using future information
  • Produce multi-step forecasts with the recursive, direct and multi-output strategies
  • Validate a model walk-forward and recognise leakage and look-ahead bias
  • Explain ridge, lasso, regression trees, random forest, gradient boosting, the MLP and the LSTM, with their main tuning parameters
  • Build prediction intervals by quantile regression and conformal prediction
  • Compare ML forecasts with statistical benchmarks on the same test period, with MASE and the Diebold–Mariano test

Reading and tools

Slide 5 in the PDF

The course textbook is Huang and Petukhina (2022), whose Chapter 10 covers machine learning for time series. The free online textbook of Hyndman and Athanasopoulos (2021), Section 12.4, presents neural network autoregressions, and Hastie, Tibshirani and Friedman (2009) give the theory of the methods: linear models and their penalties (Chapter 3), trees (Chapter 9), boosting (Chapter 10), neural networks (Chapter 11) and random forests (Chapter 15). Three reviews complete the reading: Hewamalage, Bergmeir and Bandara (2021) on recurrent networks for forecasting, Lim and Zohren (2021) on deep learning for time series and Montero-Manso and Hyndman (2021) on global models.

The computations use Python. The scikit-learn library provides every model of the chapter except the LSTM: RidgeCV and LassoCV for penalised regressions, RandomForestRegressor, HistGradientBoostingRegressor, MLPRegressor and TimeSeriesSplit for walk-forward validation. The LSTM is a small network written in PyTorch. The models are kept small, with fixed random seeds, so that every notebook runs on an ordinary processor in Google Colab in a few minutes.

In brief

After comparing the post-sample accuracy of popular ML methods with that of eight traditional statistical ones, we found that the former are dominated across both accuracy measures used and for all forecasting horizons examined.

— Makridakis, Spiliotis and Assimakopoulos (2018a), Statistical and Machine Learning forecasting methods: Concerns and ways forward

Machine learning (ML) methods such as random forests, gradient boosting and neural networks learn a prediction rule from examples, without a model written down in advance. They dominate image recognition, translation and many prediction tasks on large tabular data sets. The question of this chapter is whether they also forecast time series better than the statistical models of the previous chapters: ARIMA, exponential smoothing (ETS) and dynamic harmonic regression, and even the naive forecast.

The answer depends on the data and on the way the comparison is made. A time series has to be turned into a table of examples before any ML method can use it, and the examples are ordered and dependent. A careless table or a careless validation scheme lets information from the future into the training data, and the resulting scores can look excellent while the model has no forecasting ability. A large part of the chapter is therefore about building the table and validating the model honestly.

The chapter then applies ridge and lasso regressions, regression trees and their ensembles, small neural networks and the LSTM to four problems: the daily electricity load of Romania, Romanian inflation, the volatility of stock indices and the sign of daily returns. The ML forecasts enter the same comparison as the statistical models of Chapter 4, with the same forecast origins, the same accuracy measures and the Diebold–Mariano test. On the electricity load, a ridge regression with calendar features is as accurate as the best statistical model, while the LSTM is clearly less accurate. The landmark M4 and M5 competitions and the study of Gu, Kelly and Xiu (2020) show where ML methods do win: with many related series and rich features.

Daily electricity load of Romania, 26 forecast origins from 30 June 2025 to 15 June 2026, horizons of 1 to 14 days. Left: the mean absolute scaled error (MASE) of five statistical models (blue) and six ML models (red); below 1 means more accurate than the in-sample seasonal naive forecast. Right: the mean absolute error (MAE) in GW by horizon for six models.
Daily electricity load of Romania, 26 forecast origins from 30 June 2025 to 15 June 2026, horizons of 1 to 14 days. Left: the mean absolute scaled error (MASE) of five statistical models (blue) and six ML models (red); below 1 means more accurate than the in-sample seasonal naive forecast. Right: the mean absolute error (MAE) in GW by horizon for six models.

What you will be able to do

  • Build a supervised-learning table of lags, rolling statistics and calendar features from a time series, without future information.
  • Produce multi-step forecasts with the recursive, direct and multi-output strategies.
  • Validate a model walk-forward and recognise leakage and look-ahead bias.
  • Explain ridge, lasso, regression trees, random forests, gradient boosting, the multilayer perceptron and the LSTM, with their tuning parameters.
  • Construct prediction intervals by quantile regression and split conformal prediction and check their coverage.
  • Compare ML forecasts with statistical benchmarks on the same test period, with MASE and the Diebold–Mariano test.

Chapter plan

  1. Forecasting as supervised learningLag, rolling-window and calendar features; recursive, direct and multi-output forecasts.
  2. Validation without leakageTraining, validation and test samples, walk-forward validation, leakage and the bias–variance trade-off.
  3. Regularised linear models: ridge and lassoPenalised least squares on many correlated lags.
  4. Trees, random forest and gradient boostingRegression trees, their ensembles, extrapolation and permutation importance.
  5. Neural networksThe multilayer perceptron, its training, the recurrent network and the LSTM.
  6. Application: electricity load against Chapter 4Eleven models on 26 origins and 14 horizons, and the Easter window.
  7. Prediction intervalsQuantile regression, the pinball loss and split conformal prediction.
  8. Local and global models: Romanian inflationOne model per series against one model for 27 countries.
  9. Applications in finance: volatility and the sign of returnsHAR against ML for realised volatility; the sign of tomorrow's return.
  10. Landmark case studies: M4, M5 and Gu, Kelly and XiuForecasting competitions and machine learning for stock returns.
  11. Possible contribution of AIFoundation models, what an assistant can draft and what must be checked.
  12. SummaryKey ideas, formulas and self-assessment questions.

Forecasting as supervised learning

Supervised learning is the part of machine learning that learns to predict a target from inputs, using examples where both are known. A forecast fits this frame once the series is rewritten as a table: each row holds what was known at a forecast origin and the value that was observed later. The section builds this table, shows it on the daily electricity load of Romania and presents the three ways of producing forecasts more than one step ahead.

Supervised learning: data, loss function and training

Slide 6 in the PDF

Supervised learning starts from \(n\) pairs \((\mathbf{x}_i, y_i)\). The vector \(\mathbf{x}_i\) holds the features, the inputs of the prediction, and \(y_i\) is the target, the quantity to be predicted. The goal is a function \(\hat f\) that predicts well on new data, that is, whose expected loss \(E[L(y, \hat f(\mathbf{x}))]\) is small for pairs not used in training. The loss function decides what "predicts well" means. The squared error \((y - \hat y)^2\) is minimised by the conditional mean of \(y\), so it leads to forecasts of the mean; the absolute error \(|y - \hat y|\) is minimised by the conditional median.

Training chooses \(\hat f\) within a class of functions (linear functions, trees, neural networks) by minimising the average loss on the training data. Each class also has hyperparameters, settings fixed before training: the depth of a tree, the number of trees, the size of a penalty. They cannot be estimated from the training loss, because a more complex model always fits the training data better; they are chosen by validation on data the model has not seen.

Standard machine learning treats the pairs as independent draws from one distribution. The rows of a time-series table are ordered in time and dependent: neighbouring rows share most of their information. The table must therefore be built so that each row uses only the past of its origin, and the validation must keep the order of time. These two requirements shape the rest of the chapter.

In brief

  • Data: pairs \((\mathbf{x}_i, y_i)\), \(i = 1, \dots, n\); \(\mathbf{x}_i\): the features (inputs), \(y_i\): the target
    • goal: a function \(\hat f\) with small expected loss \(E[L(y, \hat f(\mathbf{x}))]\) on new data
    • loss: squared error \(L = (y - \hat y)^2\) (forecasts the mean), absolute error \(|y - \hat y|\) (the median)
  • Training: choose \(\hat f\) in a class of functions (linear, trees, networks) by minimising the average loss on the training data
    • hyperparameters: settings fixed before training (tree depth, number of trees, penalty); chosen by validation
  • Standard ML assumes the pairs are independent; in a time series they are ordered and dependent, so the table and the validation must respect time

From a series to a table

Slide 7 in the PDF

The simplest features are lags. At the forecast origin \(t\) the values \(y_t, y_{t-1}, \dots, y_{t-p+1}\) are known, and the target is \(y_{t+h}\), the value \(h\) steps later. An AR(\(p\)) model of Chapter 2 is exactly a linear regression of \(y_{t+1}\) on these lags. Machine learning keeps the table and replaces the linear function by a more flexible one, such as a tree ensemble or a neural network.

Rolling-window features summarise the recent past: means, standard deviations or minima over the last \(w\) values. The 7-day mean, for example, is \(\bar y_t^{(7)} = \frac{1}{7}\sum_{j=0}^{6} y_{t-j}\), the average of the seven values up to and including the origin. The window must end at \(t\). A window that contains \(y_{t+1}\) puts part of the target into the features, an error called leakage that the next section examines in detail.

Calendar features describe the target day: the day of the week, the month, the public holidays and Fourier terms such as \(\sin(2\pi k\,d/365.25)\), where \(d\) is the day of the year and \(k = 1, 2, \dots\) is the harmonic (\(k = 1\) gives one cycle per year). They are known in advance, so they are legitimate inputs, as the Fourier terms and the Orthodox Easter dummy of Chapter 4 were. Exogenous variables such as temperature are allowed only in the form available at the origin: the weather forecast, not the temperature measured later.

In brief

  • Lag features: \(y_{t}, y_{t-1}, \dots, y_{t-p+1}\), known at the forecast origin \(t\); target \(y_{t+h}\)
    • an AR\((p)\) (Chapter 2) is exactly a linear regression on this table; ML replaces the linear function by a flexible one
  • Rolling-window features: means, standard deviations, minima over the last \(w\) values
    • for example the 7-day mean \(\bar y_t^{(7)} = \frac{1}{7}\sum_{j=0}^{6} y_{t-j}\)
    • the window must end at \(t\), the origin: a window that contains \(y_{t+1}\) leaks the target
  • Calendar features of the target day: day of week, month, Fourier terms \(\sin(2\pi k\,d/365.25)\), holidays
    • \(d\): the day of the year; \(k = 1, 2, \dots\): the harmonic (one annual cycle for \(k = 1\))
    • known in advance, so allowed (Chapter 4: Fourier terms and Orthodox Easter); exogenous variables (temperature) only if their values are known or forecast at \(t\)

Worked example: a lag table for the electricity load

Slide 8 in the PDF

The table shows six rows built from the daily mean load of Romania, in gigawatts (GW), for the week that ends just before 30 June 2025. The first row has its origin on Monday 23 June: the load that day was 5.80 GW, on Sunday 4.73 GW and on Saturday 5.21 GW, and the target is Tuesday's load, 6.03 GW. Each row is a snapshot of what was known at the origin, paired with the value that the forecast had to predict.

The construction loses a few observations at both ends. With \(p\) lags, the first \(p - 1\) days have no complete row, because some of their lags fall before the start of the series. For horizon \(h\), the last \(h\) days have no observed target. A series of length \(n\) therefore gives \(n - p - h + 1\) complete rows.

The weekday column shows why calendar features matter. The Sunday load is about 1 GW below the working-day load, and the target of the last row, Sunday 29 June, is 5.09 GW after 5.78 GW on Saturday. A model that sees only the lags cannot know that the next day is a Sunday; it needs the weekday of the target day as a feature.

In brief

Origin \(t\)\(y_t\)\(y_{t-1}\)\(y_{t-2}\)target \(y_{t+1}\)
Mon5.804.735.216.03
Tue6.035.804.736.38
Wed6.386.035.806.61
Thu6.616.386.036.59
Fri6.596.616.385.78
Sat5.786.596.615.09
  • Daily mean load of Romania (GW), the last days before 30 June 2025
    • each row: what we knew at \(t\), and the value we want for \(t+1\)
  • With \(p\) lags, the first \(p - 1\) days have no complete row
    • a series of length \(n\) gives \(n - p - h + 1\) rows for horizon \(h\)
  • Weekday column: the Sunday dips are visible; a model needs the weekday of the target day

Electricity load: the series and three features

Slide 9 in the PDF

Worked example

      Question for the room With p = 5 lags and h = 3, how many complete rows do the 9 days give?
      Answer

      9 − 5 − 3 + 1 = 2 rows, with origins 25 and 26 June 2025. On the whole series the table for h = 3 has 1642 − 28 − 3 + 1 = 1612 rows.

        Question for the room Move the horizon to h = 7. What do the correlations of the target with yt and with the 7-day mean become?
        Answer

        0.81 with yt (0.77 at h = 1) and 0.60 with the 7-day mean (0.74 at h = 1). At h = 7 today is the same weekday as the target, so yt takes the role of the weekly lag, while the 7-day mean carries no weekday.

        The top panel shows the daily mean load of Romania from the European Network of Transmission System Operators for Electricity (ENTSO-E), from 1 January 2022 to 30 June 2026, 1,642 days. The red dots mark public holidays. The series combines a weekly cycle, visible as the dense oscillation, an annual cycle with peaks in winter, and sharp drops on holidays, especially around Christmas and Easter.

        The three bottom panels plot tomorrow's load \(y_{t+1}\) against three candidate features: today's load \(y_t\), the load on the same weekday of the previous week \(y_{t-6}\) and the 7-day mean at \(t\). Each dot is one row of the table; the orange dots are rows whose target falls on a Sunday and the blue ones targets from Monday to Saturday. A good feature makes the cloud narrow and close to a line.

        In brief

        • Daily mean load of Romania, ENTSO-E (the European Network of Transmission System Operators for Electricity), 1 January 2022–30 June 2026, 1 642 days; bottom: tomorrow's load against three features
        TSA_ch9_supervised_learning

        Interpreting the load features

        Slide 10 in the PDF

        The mean load over the sample is 6.18 GW. The minimum, 3.79 GW, occurred on 20 April 2025, the Orthodox Easter Sunday, and the maximum, 8.15 GW, on 13 January 2022, during a winter week. The weekly cycle, the annual cycle of heating and cooling and the holiday drops together form the multiple seasonality studied in Chapter 4.

        The correlations measure how much each feature says about tomorrow's load: 0.77 for today's load, 0.82 for the same weekday of the previous week and 0.74 for the 7-day mean. The same-weekday lag is the best single feature because it carries the weekly cycle with it. With today's load alone, the Sunday targets form a separate cloud below the others, so a single linear function of \(y_t\) cannot represent the weekly pattern; a model needs either the weekday or the right lag.

        The table used in the rest of the chapter has 1,614 rows and 30 features: 14 lags, two rolling means, the same-weekday value and calendar features (weekday dummies, holiday flags and Fourier terms). The first rows are lost to the 14 lags, which explains why there are fewer rows than days.

        In brief

        • Mean 6.18 GW; minimum 3.79 GW on 20 April 2025 (Orthodox Easter), maximum 8.15 GW on 13 January 2022 (a winter week)
          • weekly cycle, annual cycle (winter heating, summer cooling) and deep holiday dips: the multiple seasonality of Chapter 4
        • Correlation with tomorrow's load: \(y_t\) 0.77; the same weekday last week 0.82; the 7-day mean 0.74
          • with \(y_t\) alone the Sundays form a separate cloud: a single linear lag cannot represent the weekly pattern
          • the table used below has 1 614 rows and 30 features (14 lags, two means, the same weekday, calendar)

        Multi-step forecasts: three strategies

        Slide 11 in the PDF

        A forecast for several days ahead, \(h = 1, \dots, H\), can be produced in three ways. The recursive, or iterated, strategy trains one model for \(h = 1\). Its forecast \(\hat y_{T+1}\) replaces the unknown \(y_{T+1}\) in the lags, and the same model is applied again to obtain \(\hat y_{T+2}\), and so on. ARIMA models forecast in this way (Chapter 3). The strategy needs a single model, but if the one-step model is wrong, its errors are fed back as inputs and accumulate with the horizon.

        The direct strategy trains a separate model \(\hat f_h\) for each horizon, with \(y_{t+h}\) as the target. No forecast is fed back, so errors do not accumulate. The cost is \(H\) models instead of one, fewer complete rows for long horizons, and forecasts for neighbouring horizons that come from different models and need not form a coherent path.

        The multi-output strategy (MIMO, multiple-input multiple-output) trains one model that returns the whole vector \((\hat y_{T+1}, \dots, \hat y_{T+H})\) at once. It suits neural networks, whose output layer can have \(H\) units. Ben Taieb et al. (2012) review the three strategies and compare them on the NN5 forecasting competition.

        In brief

        • Recursive (iterated): one model for \(h = 1\); its forecast replaces the unknown \(y_{T+1}\) and the model is applied again
          • this is how ARIMA forecasts (Chapter 3); errors accumulate when the one-step model is wrong
        • Direct: a separate model \(\hat f_h\) for each horizon \(h = 1, \dots, H\), trained on the target \(y_{t+h}\)
          • no error accumulation, but \(H\) models and fewer rows for long horizons; the forecasts of different \(h\) need not be coherent
        • Multi-output (MIMO, multiple-input multiple-output): one model returns the vector \((\hat y_{T+1}, \dots, \hat y_{T+H})\)

        Recursive, direct and multi-output forecasts

        Slide 12 in the PDF

        Worked example

        \[\hat y_{T+1}=a+b\,y_T,\quad \hat y_{T+2}=a+b\,\hat y_{T+1}=a(1+b)+b^2y_T;\qquad \text{direct: }\hat y_{T+2}=2.3+0.6\,y_T\]
            Question for the room Set yT = 7. Which two-step forecast is larger?
            Answer

            The recursive one: 2.16 + 0.64·7 = 6.64, against 2.3 + 0.6·7 = 6.50 for the direct one. The two rules cross at yT = 3.5; above it the recursive rule, with the steeper slope, always gives more.

              Question for the room Choose the origin 6 April 2026, with Easter in the window. Which strategy has the lower MAE over the 14 days?
              Answer

              The direct one: 668 MW against 898 MW for the recursive one. At h = 7 the direct coefficient on yT is 0.81, the implied b⁷ only 0.15: iterating an AR(1) loses the weekly pattern, which the direct model keeps.

              The diagram shows the three strategies for three steps ahead. Blue boxes are inputs known at the origin \(T\), green boxes are fitted models and red boxes are forecasts. In the recursive strategy on the left, the same model \(f\) is applied three times, and the orange arrows show each forecast fed back as an input for the next step.

              In the direct strategy in the middle, the same inputs go to three different models, \(f_1\), \(f_2\) and \(f_3\), each trained for its own horizon. In the multi-output strategy on the right, one model \(f\) produces the three forecasts together. The diagram makes the trade-off visible: the recursive strategy reuses one model and propagates its errors, the direct strategy multiplies the models, and MIMO keeps one model with several outputs.

              In brief

              • Blue: the inputs known at the origin \(T\); green: the fitted models; red: the forecasts; orange: a forecast fed back as an input
              TSA_ch9_supervised_learning

              Worked example: recursive and direct two-step forecasts

              Slide 13 in the PDF

              Suppose that the one-step model estimated on the table is \(\hat y_{t+1} = 1.2 + 0.8\,y_t\) and that the last observation is \(y_T = 5\). The recursive forecast for tomorrow is \(\hat y_{T+1} = 1.2 + 0.8 \cdot 5 = 5.20\). For the day after, the unknown \(y_{T+1}\) is replaced by 5.20, which gives \(\hat y_{T+2} = 1.2 + 0.8 \cdot 5.20 = 5.36\). Substituting the first equation into the second shows the implied two-step rule: \(\hat y_{T+2} = 1.2 + 0.8(1.2 + 0.8\,y_T) = 2.16 + 0.64\,y_T\).

              The direct strategy estimates a separate regression with \(y_{t+2}\) as the target. Suppose it gives \(\hat y_{t+2} = 2.3 + 0.6\,y_t\), so the direct forecast is \(\hat y_{T+2} = 2.3 + 0.6 \cdot 5 = 5.30\). If the series really were an AR(1), the direct regression would estimate the same coefficients as the implied rule, 2.16 and 0.64, up to sampling error.

              Different coefficients signal that the one-step model is misspecified at two steps, for example because the series has a weekly pattern that one lag cannot carry. The direct model adapts to the horizon it is trained for, while the recursive model extrapolates a one-step rule. Neither strategy is always more accurate, so the choice is made on validation data.

              In brief

              • One-step model estimated on the table: \(\hat y_{t+1} = 1.2 + 0.8\,y_t\); last value \(y_T = 5\)
                • recursive: \(\hat y_{T+1} = 1.2 + 0.8 \cdot 5 = 5.20\);   \(\hat y_{T+2} = 1.2 + 0.8 \cdot 5.20 = 5.36\)
                • implied two-step rule: \(\hat y_{T+2} = 2.16 + 0.64\,y_T\)
              • Direct model estimated on the target \(y_{t+2}\): \(\hat y_{t+2} = 2.3 + 0.6\,y_t\);   \(\hat y_{T+2} = 5.30\)
                • if the AR(1) were the true model, the direct coefficients would estimate the same \(2.16\) and \(0.64\)
                • they differ: the one-step model is misspecified at two steps (for example, a weekly pattern), and the direct model adapts to the horizon
              • Neither strategy wins always: try both on the validation data

              Recap: Forecasting as supervised learning

              Slide 14 in the PDF

              A forecast is a prediction of \(y_{t+h}\) from features known at the origin \(t\): lags, rolling statistics and calendar variables of the target day. An AR(\(p\)) is a linear regression on lags, so machine learning changes the function, not the table. Forecasts several steps ahead are recursive (one model, iterated), direct (one model per horizon) or multi-output (one model with \(H\) outputs).

              In brief

              • A forecast is a prediction of \(y_{t+h}\) from features known at \(t\): lags, rolling statistics, calendar
              • An AR\((p)\) is linear regression on lags; ML changes the function, not the table
              • Multi-step: recursive (one model, iterated), direct (one model per \(h\)), MIMO (one model, \(H\) outputs)

              Section summary

              Turning a series into a table is what makes machine learning applicable to forecasting, and it is also where the first errors can enter. Every feature must be computable at the origin, the calendar of the target day must be included explicitly, and the strategy for several horizons is a modelling choice to be validated like any other.

              Validation without leakage

              A forecasting method is only as credible as the way it is evaluated. The section defines the training, validation and test samples for time series, compares four validation schemes, describes leakage, the most common source of false results, and shows it on a random walk and on the S&P 500. It ends with the bias–variance decomposition, which explains why the complexity of a model must be chosen on data it has not seen.

              Training, validation and test samples

              Slide 15 in the PDF

              The data have three roles. The training sample estimates the parameters of a model, the validation sample chooses its hyperparameters, and the test sample measures the final accuracy. The test sample is used once, at the end: if it is used to choose among models, it becomes a second validation sample and its error is no longer an honest estimate. For a time series the three samples are consecutive blocks in this order, so that every model is evaluated on a period that comes after the data it learned from.

              Walk-forward validation, the time-series cross-validation of Chapter 4, repeats this split. The model is trained on the data up to an origin, forecasts the next block, and the origin moves forward. With an expanding window the training sample contains all past data; with a rolling window it contains only the last \(W\) observations, which adapts faster to changes in the series but uses less data. The errors of all origins are pooled into the MAE, the root mean squared error (RMSE) and the MASE of Chapter 0, and two methods are compared with the Diebold–Mariano test (Diebold and Mariano, 1995).

              Random \(K\)-fold cross-validation splits the rows at random into \(K\) groups and uses each group once as test data. Bergmeir, Hyndman and Koo (2018) show that it remains valid for purely autoregressive models whose errors are white noise, because the rows are then conditionally uncorrelated. When the errors are correlated, or the targets overlap, random folds give optimistic errors.

              In brief

              • Training sample: estimates the parameters; validation: chooses the hyperparameters; test: measures the final accuracy, used once
                • for time series the three samples are consecutive blocks, in this order
              • Walk-forward validation (time-series cross-validation, Chapter 4): train on data up to an origin, forecast the next block, move the origin forward
                • expanding window: all past data; rolling window: the last \(W\) observations (adapts to change, uses less data)
                • the errors of all origins are pooled into MAE, RMSE, MASE (Chapter 0) and compared with the Diebold–Mariano test
              • Random \(K\)-fold cross-validation (random folds) is valid only for pure autoregressions with white-noise errors Bergmeir, Hyndman and Koo (2018)

              Four validation schemes

              Slide 16 in the PDF
                Question for the room Move the gap to h = 6. What share of the test rows of walk-forward without a gap has a training row less than 6 steps away?
                Answer

                63% (25% at h = 3); 100% in random K-fold, 0% with a gap. The first h − 1 = 5 of the 8 rows of each test block have targets that overlap the last training rows.

                Each panel shows four splits of the same series along the time axis: blue cells are training data, red cells test data, golden cells a gap and empty cells data that are not used. In random \(K\)-fold the test cells are scattered through the sample, so the model is often trained on observations that come after the test day: it learns from the future.

                The three walk-forward schemes keep the order of time. With an expanding window the training block grows from one split to the next; with a rolling window it has a fixed length and slides forward. In walk-forward validation with a gap, the training rows just before each test block are dropped. These rows have targets that extend into the test period, for example a 21-day return that starts a few days before the test block, and the gap prevents them from sharing information with the test rows.

                In brief

                • Random \(K\)-fold mixes past and future; walk-forward keeps the order; a gap removes the training rows whose targets overlap the test block
                TSA_ch9_validation

                Look-ahead bias and leakage

                Slide 17 in the PDF

                Leakage is the use, in training or in the features, of information that would not have been available at the forecast origin (Kaufman et al., 2012). Its effect is always the same: validation scores that look excellent and disappear when the model is used for real forecasts. In finance the same error is called look-ahead bias, because the backtest looks ahead at data that the investor could not have known.

                The typical sources are easy to miss. A rolling mean or a difference that includes \(y_{t+1}\) contains part of the target; centred moving averages and two-sided filters, such as those used to estimate a trend in Chapter 0, use future values by construction. Scaling, imputation of missing values or feature selection estimated on the whole sample before the split pass information from the test period into training. Overlapping targets, such as sums over the next \(h\) days, leak through random folds or through walk-forward splits without a gap. Revised macroeconomic data are a subtler case: the GDP figure published today for a past quarter was not known at the time, and the actual weather is not the weather forecast that was available.

                The rule that prevents leakage is to write every feature as a function of the data up to \(t\) only, and to refit every preprocessing step, such as the mean and standard deviation used for scaling, inside each training window.

                In brief

                • Leakage: information that would not be available at the forecast origin enters training or the features Kaufman et al. (2012)
                  • result: excellent validation scores that disappear in real use; look-ahead bias is its name in finance
                • Typical sources
                  • a rolling mean or a difference that includes \(y_{t+1}\); centred moving averages and two-sided filters (Chapter 0)
                  • scaling, imputation or feature selection fitted on the whole sample before the split
                  • overlapping targets (sums over the next \(h\) days) with random folds or without a gap
                  • revised macro data: today's GDP vintage was not known at the origin; actual weather instead of the weather forecast
                • Rule: write the feature code as a function of the data up to \(t\) only, and refit every preprocessing step inside each training window

                Leakage in practice: random folds on overlapping targets

                Slide 18 in the PDF

                Worked example

                \[\rho(Y_t,Y_{t+k})=\max\!\Big(0,1-\frac{k}{h}\Big),\qquad R^2=1-\frac{\sum_{\text{folds}}\sum_t(y_t-\hat y_t)^2}{\sum_{\text{folds}}\sum_t(y_t-\bar y_{\text{train}})^2}\]
                    Question for the room Choose h = 63. What R² do random 5-fold and walk-forward with a gap give for the S&P 500?
                    Answer

                    43.4% and −43.6%. Neighbouring targets share 62 of 63 returns (sample correlation 0.976), so random folds leak even more. Without a gap walk-forward gives −36.2%: the last training rows have targets that overlap the first test rows, a small leak that the gap removes.

                      Question for the room Choose h = 1: the target is one return, without overlap. What R² does random 5-fold report for the simulated random walk?
                      Answer

                      −4.5%, close to walk-forward (−6.3%) and to walk-forward with a gap (−6.1%). Without overlapping targets a test row has no near-copies in the training folds, and the 34.5% disappears.

                      The experiment shows how strong leakage can be. A random forest predicts the sum of the next 21 daily returns, roughly one month ahead, from past sums over 5, 21 and 63 days and from past volatilities. The left pair of bars uses a simulated random walk, for which the future sum is unpredictable by construction; the right pair uses 6,631 daily returns of the S&P 500 from 2000 to 2026.

                      The vertical axis is the out-of-sample \(R^2\), \(1 - \sum(y - \hat y)^2/\sum(y - \bar y)^2\), in percent. It is positive when the model forecasts better than the historical mean of the target and negative when it forecasts worse. Red bars come from random 5-fold cross-validation, blue bars from walk-forward validation and green bars from walk-forward validation with a gap of 21 days.

                      In brief

                      • Random forest for the sum of the next 21 daily returns; features: past sums over 5, 21, 63 days and volatilities; left: a simulated random walk; right: the S&P 500, 6 631 days
                      TSA_ch9_validation

                      Interpreting the leakage experiment

                      Slide 19 in the PDF

                      Random 5-fold cross-validation reports an \(R^2\) of 34.5% for the random walk, where nothing is predictable, and 36.8% for the S&P 500. Both numbers are artefacts. The target of day \(t\) and the target of day \(t+1\) share 20 of their 21 daily returns, so almost every test row has near-copies among the training rows of the other folds, and the forest simply recognises them.

                      Walk-forward validation gives \(-26.5\)% for the random walk and \(-38.6\)% for the S&P 500; with a gap of 21 days the values are \(-26.8\)% and \(-38.9\)%. A negative out-of-sample \(R^2\) means that the forest forecasts worse than the historical mean, which is the expected result for future returns (Chapter 1). The gap changes little here because the walk-forward blocks are long, but it removes the remaining overlap at their borders.

                      The lesson is general: a model is only as good as its validation, and a spectacular score on a problem known to be hard is a reason to look for a leak before believing it.

                      In brief

                      • Random 5-fold reports \(R^2\) = 34.5% for a random walk, where nothing is predictable, and 36.8% for the S&P 500
                        • neighbouring targets share 20 of 21 returns: a test row has its near-copies in the training folds
                      • Walk-forward: \(-\)26.5% and \(-\)38.6%; with a 21-day gap: \(-\)26.8% and \(-\)38.9%
                        • negative out-of-sample \(R^2\): the forest is worse than the historical mean, as expected for returns (Chapter 1)
                      • A model is only as good as its validation; a spectacular score is a reason to look for a leak

                      Overfitting and the bias–variance trade-off

                      Slide 20 in the PDF

                      Suppose the data follow \(y = f(x) + \varepsilon\), where \(f\) is the true function and \(\varepsilon\) is noise with variance \(\sigma^2\), and that \(\hat f\) is estimated on a random training sample. For a new observation \(y_0\) at a point \(x_0\), the expected squared error, taken over training samples and noise, splits into three parts:

                      \[\begin{aligned} E[(y_0 - \hat f(x_0))^2] &= \bigl(E[\hat f(x_0)] - f(x_0)\bigr)^2 \\ &\quad + \Var(\hat f(x_0)) + \sigma^2 . \end{aligned}\]
                      (9.1)

                      The first term is the squared bias, the systematic error of the method averaged over training samples. The second is the variance, which measures how much the fitted function changes from one training sample to another. The third, \(\sigma^2\), is the noise that no model can remove.

                      A rigid model, such as a very shallow tree, has a large bias: it misses part of the signal, which is called underfitting. A very flexible model has a small bias but a large variance: it follows the noise of its particular training sample, which is called overfitting, and shows a small training error with a large test error. Every ML method has a parameter that moves it along this trade-off, such as the tree depth, the number of trees, the penalty or the number of hidden units, and validation sets this parameter.

                      In brief

                      • Model \(y = f(x) + \varepsilon\), \(\Var(\varepsilon) = \sigma^2\); \(\hat f\) estimated on a random training sample
                        • the expected test error at a point \(x_0\):
                        • \(E[(y_0 - \hat f(x_0))^2] = \underbrace{(E[\hat f(x_0)] - f(x_0))^2}_{\text{bias}^2} + \underbrace{\Var(\hat f(x_0))}_{\text{variance}} + \sigma^2\)
                        • \(f\): the true function; \(y_0\): a new observation at \(x_0\); the expectation is over training samples and noise; \(\sigma^2\): the noise that no model can remove
                      • Overfitting: a flexible model learns the noise; small training error, large test error
                        • underfitting: a rigid model misses the signal (large bias)
                      • Every ML method has a complexity knob (depth, number of trees, penalty, hidden units); validation sets it

                      Bias and variance of regression trees

                      Slide 21 in the PDF
                        Question for the room Raise the noise to σ = 0.6. At what depth is the expected test error smallest?
                        Answer

                        At depth 2 (0.465; the noise variance is 0.36). With σ = 0.1 the minimum moves to depth 6. More noise raises the variance of deep trees, so the right complexity falls.

                        The simulation makes (9.1) visible. The true function is \(f(x) = \sin(2\pi x)\), the noise has standard deviation \(\sigma = 0.3\), so \(\sigma^2 = 0.09\), and each training sample has 80 points. Regression trees of depth 1 to 10 are fitted to 300 independent training samples, which allows the bias and the variance to be computed exactly as averages over samples.

                        The left panel shows one training sample (dots), the true function (black) and three fitted trees. The tree of depth 1 is a single step and misses the shape of the sine. The tree of depth 3 follows the shape with a few steps. The tree of depth 10 chases individual points and jumps around the true curve. The right panel plots, against the depth, the squared bias, the variance, their sum plus the noise (the expected test error, red), the training error (dashed) and the noise variance (dotted).

                        In brief

                        • Simulation: \(y = \sin(2\pi x) + \varepsilon\), \(\sigma = 0.3\), 80 points, 300 training samples; trees of depth 1 to 10
                        TSA_ch9_validation

                        Interpreting the U-shaped test error

                        Slide 22 in the PDF

                        At depth 1 the squared bias is 0.070 and the variance 0.031; at depth 10 the squared bias is practically 0 and the variance 0.090. As the depth grows, the bias falls and the variance rises, so the expected test error first decreases and then increases, a U shape. Its minimum, 0.132, is at depth 3; it lies above the noise variance of 0.09, which is the floor that no model can go below.

                        The training error behaves differently: it falls steadily and reaches 0.002 at depth 10, far below the noise variance, because the deep tree has memorised its sample. The training error therefore cannot choose the complexity; only data that the model has not seen can. The ensembles of the next sections (random forests and boosting) and the penalties of ridge and lasso are ways of reducing the variance without giving up much of the bias.

                        In brief

                        • Depth 1: bias\(^2\) 0.070, variance 0.031; depth 10: bias\(^2\) 0.000, variance 0.090
                          • the expected test error is smallest at depth 3 (0.132), above the noise variance 0.09
                        • The training error falls to 0.002 at depth 10: it cannot choose the complexity
                        • Ensembles (random forest, boosting) and penalties (ridge, lasso) reduce the variance without losing much bias

                        Recap: Validation without leakage

                        Slide 23 in the PDF

                        Walk-forward validation trains on the past, tests on the next block and moves forward; random folds are acceptable only for pure autoregressions with white-noise errors. Leakage enters through future values in features, preprocessing fitted on the full sample, overlapping targets and revised data. The test error is the sum of the squared bias, the variance and the noise, and validation sets the complexity that balances the first two.

                        In brief

                        • Walk-forward: train on the past, test on the next block, move on; random folds only for pure autoregressions
                        • Leakage: future values in features, preprocessing on the full sample, overlapping targets, revised data
                        • Test error = bias\(^2\) + variance + noise; validation sets the complexity

                        Section summary

                        Honest validation keeps the order of time, refits every step of the analysis inside each training window and separates the rows whose targets overlap. On overlapping 21-day returns, random folds turn a negative out-of-sample \(R^2\) into an apparent 35%, even on a random walk. The bias–variance trade-off explains why complexity must be tuned on unseen data, and why the methods that follow all come with a complexity parameter.

                        Regularised linear models: ridge and lasso

                        Before nonlinear methods, the section returns to linear regression on lags, the AR model of Chapter 2 with many lags and calendar features. With 30 correlated features, ordinary least squares becomes unstable; ridge and lasso add a penalty on the size of the coefficients, which trades a little bias for a large reduction in variance. The section derives both estimators, works through a one-feature example and shows their coefficient paths on the electricity load.

                        Many lags, unstable least squares

                        Slide 24 in the PDF

                        The linear model on \(p\) features predicts \(\hat y = \beta_0 + \sum_j \beta_j x_j\), and ordinary least squares (OLS) chooses the coefficients that minimise the sum of squared errors \(\sum_t (y_t - \hat y_t)^2\). When the features are 14 lags of a daily series, neighbouring lags are strongly correlated: yesterday's load and the load of the day before carry almost the same information. OLS can then trade a large positive coefficient on one lag against a large negative one on its neighbour with little change in the fit, so the coefficients become large, of opposite signs, and change from one training window to the next.

                        Penalised regressions measure the size of the coefficients, and this size depends on the units of the features. The features are therefore standardised first, \(x_j \to (x_j - \bar x_j)/s_j\), where \(\bar x_j\) and \(s_j\) are the mean and standard deviation in the training window only; computing them on the full sample would be leakage. Without standardisation, the penalty would treat a lag measured in GW and a 0/1 weekday dummy unequally.

                        The idea of regularisation follows from the bias–variance trade-off (9.1): shrinking the coefficients towards zero introduces a little bias, but it reduces the variance of the estimates much more, and the test error falls.

                        In brief

                        • Linear model on \(p\) standardised features: \(\hat y = \beta_0 + \sum_j \beta_j x_j\); OLS (ordinary least squares) minimises \(\sum_t (y_t - \hat y_t)^2\)
                          • neighbouring lags are highly correlated: OLS coefficients become large, of opposite signs and unstable from one window to the next
                        • Standardise first: \(x_j \to (x_j - \bar x_j)/s_j\), with \(\bar x_j\), \(s_j\) from the training window only
                          • otherwise the penalty treats features measured in GW and in dummies unequally
                        • Idea of regularisation: accept a little bias in exchange for a large fall in variance

                        Ridge and lasso

                        Slide 25 in the PDF

                        Ridge regression (Hoerl and Kennard, 1970) adds the sum of squared coefficients to the least-squares criterion:

                        \[\min_{\beta}\ \sum_t (y_t - \beta_0 - \mathbf{x}_t'\beta)^2 + \lambda\sum_j \beta_j^2 ,\]
                        (9.2)

                        where \(\lambda \ge 0\) is the penalty; \(\lambda = 0\) gives OLS. With \(\mathbf{X}\) the matrix of standardised features and \(\mathbf{y}\) the vector of targets (both centred, so that \(\beta_0\) drops out), the solution has the closed form \(\hat\beta = (\mathbf{X}'\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}'\mathbf{y}\), with \(\mathbf{I}\) the identity matrix. Adding \(\lambda\) to the diagonal makes the matrix well conditioned even when the lags are nearly collinear. All coefficients shrink towards 0, but none becomes exactly 0.

                        The lasso of Tibshirani (1996) (least absolute shrinkage and selection operator) uses the penalty \(\lambda\sum_j |\beta_j|\) instead. The absolute value has a corner at zero, and as a result the lasso sets some coefficients exactly to 0: it shrinks and selects features at the same time. With a single standardised feature, \(S_{xy} = \sum_t x_ty_t\) and \(S_{xx} = \sum_t x_t^2\), the two estimators have explicit forms:

                        \[\begin{aligned} \hat\beta_{\text{ridge}} &= \frac{S_{xy}}{S_{xx} + \lambda}, \\ \hat\beta_{\text{lasso}} &= \frac{\mathrm{sign}(S_{xy})\max(|S_{xy}| - \lambda/2,\ 0)}{S_{xx}} . \end{aligned}\]
                        (9.3)

                        The lasso formula is called soft thresholding. With \(S_{xy} = 90\) and \(S_{xx} = 100\), OLS gives \(90/100 = 0.90\); ridge with \(\lambda = 50\) gives \(90/150 = 0.60\); the lasso with \(\lambda = 40\) gives \((90 - 20)/100 = 0.70\); and for every \(\lambda \ge 180\) the lasso coefficient is exactly 0. The penalty \(\lambda\) is a hyperparameter, chosen by walk-forward validation (TimeSeriesSplit), never by random folds.

                        In brief

                        • Ridge Hoerl and Kennard (1970): \(\min_{\beta} \sum_t (y_t - \beta_0 - \mathbf{x}_t'\beta)^2 + \lambda\sum_j \beta_j^2\)
                          • \(\lambda \ge 0\): the penalty (\(\lambda = 0\): OLS); \(\mathbf{X}\): the matrix of standardised features; \(\mathbf{y}\): the targets; \(\mathbf{I}\): the identity matrix
                          • closed form \(\hat\beta = (\mathbf{X}'\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}'\mathbf{y}\); all coefficients shrink towards 0, none becomes exactly 0
                        • Lasso Tibshirani (1996) (least absolute shrinkage and selection operator): penalty \(\lambda\sum_j |\beta_j|\)
                          • sets some coefficients exactly to 0: it selects lags
                        • Worked example, one standardised feature, \(S_{xy} = \sum x_ty_t = 90\), \(S_{xx} = \sum x_t^2 = 100\)
                          • OLS \(90/100 = 0.90\); ridge, \(\lambda = 50\): \(90/150 = 0.60\); lasso, \(\lambda = 40\): \((90 - 20)/100 = 0.70\); lasso, \(\lambda \ge 180\): exactly 0
                          • lasso in one dimension: \(\hat\beta = \mathrm{sign}(S_{xy})\max(|S_{xy}| - \lambda/2, 0)/S_{xx}\) (soft thresholding)
                        • \(\lambda\) is chosen by walk-forward validation (TimeSeriesSplit), never by random folds

                        Ridge and lasso paths on the load features

                        Slide 26 in the PDF

                        Worked example: ridge

                        \[\hat\beta_{\text{ridge}}=\frac{S_{xy}}{S_{xx}+\lambda}=\frac{S_{xx}}{S_{xx}+\lambda}\,\hat\beta_{\text{OLS}}\]

                          Worked example: lasso

                          \[\hat\beta_{\text{lasso}}=\operatorname{sign}(S_{xy})\,\frac{\max(|S_{xy}|-\lambda/2,\,0)}{S_{xx}}\]
                              Question for the room Set Sxy = 30. From what λ is the lasso coefficient exactly 0, and what does ridge give with λ = 50?
                              Answer

                              From λ = 2·30 = 60; ridge gives 30/150 = 0.200, not zero. The lasso threshold grows with |Sxy|: features weakly related to the target leave first.

                                Question for the room Move the penalty 6 grid steps towards a larger λ (−6). How many features remain non-zero?
                                Answer

                                19 of 30, at λ = 0.00441 (22 at the chosen λ, 0.00173). The order of entry does not depend on the chosen λ: lag6, lag0, lag13.

                                  Question for the room Cut the sample at 31 December 2023. How many features does the lasso keep at the chosen penalty?
                                  Answer

                                  23 of 30 (22 on the data up to 30 June 2025), on 702 rows, with λ = 0.00486; the first features to enter are still lag6, lag0 and lag13.

                                  A coefficient path shows every coefficient as a function of the penalty. The two panels use the same problem: tomorrow's load as the target, the 30 standardised features of the load table and the data up to 30 June 2025, 1,249 rows. The penalty decreases from left to right on a logarithmic scale, so each panel starts from a heavily penalised model, where all coefficients are 0, and ends close to OLS. The vertical axis is the coefficient in GW per standard deviation of the feature.

                                  Seven features are highlighted in colour: today's load (lag0), the same weekday of the previous week (same_dow), the 7-day mean (mean7), the Saturday dummy and the Easter, Christmas and holiday flags; the thin lines are the other features. In the ridge panel all lines leave zero together and grow smoothly. In the lasso panel the lines leave zero one after another and some stay at zero over a wide range of penalties. The dashed red line marks the lasso penalty chosen by walk-forward cross-validation.

                                  In brief

                                  • Target: tomorrow's load; 30 standardised features; data up to 30 June 2025 (1 249 rows); dashed: the lasso penalty chosen by walk-forward cross-validation
                                  TSA_ch9_regularisation_trees

                                  Interpreting the coefficient paths

                                  Slide 27 in the PDF

                                  With a large penalty all coefficients are close to 0. As \(\lambda\) falls, the lasso first admits the lag of the same weekday last week, then today's load and the lag of 13 days; today's load soon dominates and ends with a coefficient of 0.55 GW per standard deviation. At the chosen penalty the next largest coefficients are those of the weekday dummies, the largest of them the Monday dummy with 0.38 GW: after the persistence of the series, the calendar carries most of the remaining signal.

                                  At the penalty chosen by validation, the lasso keeps 22 of the 30 features. Many lags become redundant once today's load, the weekday and the holidays are in the model, which is the multicollinearity of the previous slide resolved by selection. Ridge keeps all 30 features and shrinks them smoothly. On the 26 test origins of Section 6, ridge turns out to be the most accurate ML model, as accurate as the best statistical model of Chapter 4.

                                  In brief

                                  • With a large penalty all coefficients are near 0; as \(\lambda\) falls, the first to enter are lag6 (\(y_{t-6}\), the same weekday as the target), lag0 (\(y_t\)) and lag13; at the chosen penalty \(y_t\) dominates, with coefficient 0.55 GW per standard deviation
                                    • the next largest coefficients are the weekday dummies (Monday 0.38): the calendar carries most of the remaining signal
                                  • At the chosen penalty the lasso keeps 22 of 30 features: many lags are redundant once \(y_t\), the weekday and the holidays are in
                                  • Ridge keeps everything but shrinks it smoothly; on the test origins of Section 6 ridge is the best ML model

                                  Recap: Ridge and lasso

                                  Slide 28 in the PDF

                                  Ridge and lasso are penalised least squares: the ridge penalty \(\lambda\sum\beta_j^2\) shrinks all coefficients, the lasso penalty \(\lambda\sum|\beta_j|\) shrinks them and sets some to zero. The features are standardised inside each training window, and \(\lambda\) is chosen walk-forward. A regularised autoregression with calendar features is a strong baseline that every nonlinear method has to beat.

                                  In brief

                                  • Penalised least squares: ridge \(\lambda\sum\beta_j^2\) shrinks; lasso \(\lambda\sum|\beta_j|\) shrinks and selects
                                  • Standardise inside each training window; choose \(\lambda\) walk-forward
                                  • A regularised AR with calendar dummies is a strong ML baseline

                                  Section summary

                                  Penalties make a linear model with many correlated lags stable. Ridge shrinks every coefficient, the lasso also selects; on the electricity load the lasso keeps 22 of 30 features, led by today's load and the calendar. Because the linear structure is often close to right for seasonal series, a well-specified ridge regression is the benchmark for the nonlinear methods of the next sections.

                                  Trees, random forest and gradient boosting

                                  Regression trees capture nonlinearities and interactions without being told where they are, but a single tree is unstable. Random forests average many trees to reduce their variance, and gradient boosting adds small trees one at a time to reduce their bias. The section explains the three methods, compares them on the electricity load, shows the main limitation of all tree methods, their inability to extrapolate, and measures which features a fitted model uses.

                                  Regression trees

                                  Slide 29 in the PDF

                                  A regression tree partitions the feature space with a sequence of yes/no questions of the form "\(x_j \le c\)?". Each question splits a group of training rows into two, and each final group, called a leaf, predicts the mean target of its training rows. The splits are chosen greedily, one at a time: at each node the algorithm tries every feature \(j\) and every threshold \(c\) and keeps the pair that gives the smallest sum of squared errors in the two child groups.

                                  Trees have two strengths for forecasting. They capture nonlinear effects and interactions, such as a weekday effect that differs between winter and summer, without the analyst specifying them, and they need no scaling of the features, since a split depends only on the order of the values. Their weaknesses are equally clear: a single tree is unstable, because a small change in the data can change the first split and with it the whole tree, and its prediction is a step function.

                                  The most important limitation for time series is that a tree cannot extrapolate. Its forecast is always the mean of some training targets, so it can never exceed the largest target or fall below the smallest one seen in training. For trending series the remedy is to model differences, growth rates or deviations from a trend, never levels.

                                  In brief

                                  • A regression tree splits the feature space with questions “\(x_j \le c\)?” and predicts, in each final region (leaf), the mean target of its training rows
                                    • the split is chosen greedily: the \((j, c)\) with the smallest sum of squared errors of the two children
                                  • Strengths: nonlinearity and interactions (weekday \(\times\) season) without specifying them; no scaling needed
                                    • weaknesses: a single tree is unstable (high variance) and piecewise constant
                                  • A tree cannot extrapolate: its forecast is always a mean of training targets
                                    • for trending series: model differences, growth rates or the deviation from a trend, not levels

                                  A depth-2 tree for tomorrow's load

                                  Slide 30 in the PDF
                                    Question for the room Train the tree on the data up to 29 June 2026. Does the first question change?
                                    Answer

                                    The feature stays same_dow and the threshold falls from 6.08 to 6.04 GW; the leaves become 5.27, 5.86, 6.37 and 7.17 GW, with 1613 days, and R² stays 0.64.

                                    The tree is fitted to the load data up to 30 June 2025, 1,249 days, with four features: today's load (lag0), the load on the same weekday of the previous week (same_dow), and the weekend and holiday flags. Its depth is limited to 2, so it asks at most two questions, and each leaf must contain at least 20 days.

                                    Each box shows the splitting rule, the number of training days that reach the node (samples) and the mean load of those days in GW (value). The root contains all 1,249 days, with mean 6.19 GW. Days that satisfy the rule go to the left branch (True), the others to the right (False). The left branch splits again on the same-weekday load at 5.5 GW, the right branch on today's load at 6.94 GW.

                                    In brief

                                    • Features: \(y_t\) (lag0), the same weekday last week (same_dow), weekend and holiday flags; data up to 30 June 2025; at least 20 days per leaf
                                    TSA_ch9_regularisation_trees

                                    Interpreting the tree

                                    Slide 31 in the PDF

                                    The first question is whether the load on the same weekday of the previous week was below 6.08 GW. Weekends and holidays have low load, and the same day a week earlier was usually also a weekend day, so this question separates them from working days without any weekday dummy. The left branch (548 days, mean 5.67 GW) then separates the lowest days from the rest; the right branch (701 days, mean 6.59 GW) separates ordinary working days from the days of highest load.

                                    The four leaves predict 5.29, 5.89, 6.39 and 7.17 GW and contain 195, 353, 523 and 178 days. These four numbers already explain 64% of the variance of tomorrow's load in the training data (\(R^2 = 0.64\)). Deeper trees add more detail, but also more variance, as the bias–variance experiment showed. Whatever the inputs, the forecast of this tree for any day is one of the four leaf means.

                                    In brief

                                    • First question: was the same weekday last week below 6.08 GW? It separates weekends and holidays from working days
                                      • four leaves: 5.29, 5.89, 6.39 and 7.17 GW, with 195, 353, 523 and 178 days
                                    • Four numbers already explain \(R^2\) = 0.64 of the variance in training; deeper trees add detail and variance
                                    • The forecast for any day is one of these four means

                                    Random forest

                                    Slide 32 in the PDF

                                    Bagging, short for bootstrap aggregation, reduces the variance of trees by averaging. It draws \(B\) bootstrap samples of the training rows (samples of the same size, drawn with replacement), grows a deep tree on each, and averages their forecasts. If each tree has variance \(\sigma^2\) and two trees have correlation \(\rho\), the average of \(B\) trees has variance

                                    \[\Var\Bigl(\frac{1}{B}\sum_{b=1}^{B} T_b(\mathbf{x})\Bigr) = \rho\sigma^2 + \frac{1-\rho}{B}\,\sigma^2 ,\]

                                    where \(T_b\) is the \(b\)-th tree. The second term vanishes as \(B\) grows, but the first remains: averaging cannot remove the variance that the trees share.

                                    The random forest of Breiman (2001) attacks the first term. At each split, only a random subset of the features is tried (the hyperparameter max_features), so the trees differ more from one another, \(\rho\) falls, and the variance of the average falls with it. The main hyperparameters are the number of trees, where more trees never make the forest worse, only slower, the size of the feature subset and the minimum leaf size.

                                    The bootstrap draws rows without regard to their order in time. This does not violate the rules of the previous section, because the bootstrap only builds the model inside the training window, and the evaluation remains walk-forward on later data.

                                    In brief

                                    • Bagging (bootstrap aggregation): grow \(B\) deep trees on bootstrap samples of the rows and average them
                                      • averaging \(B\) trees with correlation \(\rho\) and variance \(\sigma^2\) gives variance \(\rho\sigma^2 + (1 - \rho)\sigma^2/B\)
                                    • Random forest Breiman (2001): at each split, only a random subset of features is tried (max_features)
                                      • this decorrelates the trees (\(\rho\) falls), so the average has a lower variance
                                    • Hyperparameters: number of trees (more is never worse, only slower), max_features, minimum leaf size
                                      • the bootstrap ignores time order inside the training window; that is fine, because the validation is walk-forward

                                    Gradient boosting

                                    Slide 33 in the PDF

                                    Boosting (Friedman, 2001) builds the model in the opposite way. Instead of averaging deep, independent trees, it adds small trees one after another, each fitted to the residuals of the current model. For the squared error, the residuals are the negative gradient of the loss with respect to the current forecast, which gives the method its name. After \(m\) trees the model is

                                    \[F_m(\mathbf{x}) = F_{m-1}(\mathbf{x}) + \nu\,g_m(\mathbf{x}) ,\]
                                    (9.4)

                                    where \(g_m\) is the \(m\)-th small tree and \(\nu\), the learning rate, usually between 0.01 and 0.3, scales its contribution. A small learning rate makes each step cautious, so the model needs more trees but overfits less.

                                    A worked example shows one step. The targets are 5, 6 and 8. The model starts from their mean, \(F_0 = 6.33\), so the residuals are \(-1.33\), \(-0.33\) and \(+1.67\). A tree with one split (a stump) that isolates the third point predicts the residual 1.67 for it. With \(\nu = 0.1\), the forecast for that point becomes \(F_1 = 6.33 + 0.1 \cdot 1.67 = 6.50\): it moves one tenth of the way towards the target, and the next trees continue from the new residuals.

                                    Fast implementations bin each feature into a histogram, which makes the search for splits much faster: XGBoost (Chen and Guestrin, 2016), LightGBM (Ke et al., 2017) and HistGradientBoostingRegressor in scikit-learn. In the rest of the chapter GB denotes gradient boosting; the winners of the M5 competition (Section 10) used LightGBM. The number of trees is chosen by early stopping, which stops adding trees when the error on validation data no longer falls; the other hyperparameters are the tree size, the learning rate and the minimum leaf size.

                                    In brief

                                    • Boosting Friedman (2001): add small trees one at a time, each fitted to the residuals (the negative gradient of the loss) of the current model
                                      • \(F_m(\mathbf{x}) = F_{m-1}(\mathbf{x}) + \nu\,g_m(\mathbf{x})\); \(F_m\): the model after \(m\) trees; \(g_m\): the \(m\)-th small tree; \(\nu\): the learning rate (0.01–0.3)
                                    • Worked example: targets \(5, 6, 8\); start with the mean \(F_0 = 6.33\); residuals \(-\)1.33, \(-\)0.33, +1.67
                                      • a stump that isolates the third point predicts \(1.67\) there; with \(\nu = 0.1\): \(F_1 = 6.33 + 0.1 \cdot 1.67 = 6.500\)
                                    • Fast implementations bin the features into histograms: XGBoost Chen and Guestrin (2016), LightGBM Ke et al. (2017), HistGradientBoostingRegressor in scikit-learn
                                      • below we write GB for gradient boosting; the winners of M5 (Section 10) used LightGBM
                                      • hyperparameters: number of trees with early stopping on validation data, tree size, \(\nu\), minimum leaf size

                                    One tree, a forest and boosting on the load

                                    Slide 34 in the PDF

                                    Worked example

                                    \[F_0=\bar y,\qquad r_i=y_i-F_{m-1}(x_i),\qquad F_m(x)=F_{m-1}(x)+\nu\,g_m(x)\]
                                        Question for the room Set ν = 1. What is F1 at the three points?
                                        Answer

                                        5.50, 5.50 and 8.00: the first tree fully corrects the residual of the isolated point, and the training MSE falls from 1.56 to 0.167 after one tree. A large learning rate fits the training data fast, noise included.

                                          Question for the room Train up to 31 December 2023 (546 validation days). After how many trees does boosting with ν = 0.3 reach its minimum?
                                          Answer

                                          After 24 trees (MSE 0.093), against 72 in the lecture; with ν = 0.05 the minimum comes after 222 trees (0.092). On a shorter training sample fast boosting starts fitting noise earlier.

                                          The chart compares the three methods on tomorrow's load. They are trained on the data up to 31 December 2024 (1,069 days) and evaluated on the validation period from January to June 2025 (180 days). The vertical axis is the validation mean squared error (MSE) in GW². The dotted line is the seasonal naive forecast, the load of the same weekday last week.

                                          The left panel varies the maximum depth of a single tree and of the trees of a random forest of 200 trees. The right panel shows boosting as the number of trees grows, on a logarithmic scale, for two learning rates, 0.3 and 0.05. Each curve shows how validation error responds to the complexity knob of its method.

                                          In brief

                                          • Tomorrow's load; training up to 31 December 2024 (1069 days), validation January–June 2025 (180 days); dotted: the seasonal naive forecast
                                          TSA_ch9_regularisation_trees

                                          Interpreting the ensembles

                                          Slide 35 in the PDF

                                          The best single tree reaches a validation MSE of 0.183 at depth 9, and its curve is flat and noisy beyond depth 3. The random forest reaches 0.105 at depth 12 and keeps improving with depth, because averaging removes most of the variance that deep trees add. Both are far below the seasonal naive forecast, whose MSE is 0.303.

                                          Boosting with \(\nu = 0.3\) reaches its minimum, 0.110, after 72 trees; with \(\nu = 0.05\) it needs 256 trees and reaches 0.098, the lowest value in the comparison. This is the usual pattern: a small learning rate needs more trees and ends slightly lower. Early stopping finds the number of trees automatically, at the point where the validation error stops falling.

                                          In brief

                                          • Single tree: best validation MSE (mean squared error) 0.183 at depth 9; random forest: 0.105 at depth 12
                                            • averaging removes most of the variance; the seasonal naive forecast has 0.303
                                          • Boosting with \(\nu = 0.3\): minimum 0.110 after 72 trees; with \(\nu = 0.05\): 0.098 after 256 trees
                                            • a small learning rate needs more trees and usually ends slightly lower; early stopping picks the number of trees

                                          Trees cannot extrapolate: the Romanian price level

                                          Slide 36 in the PDF
                                            Question for the room Train the forests up to December 2022. What is the RMSE on levels?
                                            Answer

                                            22.9 index points (36.9 with training up to 2020): the largest level seen is now 135.6, but the forecast stays capped at 134.1 while the index reaches 173.4. On monthly changes the RMSE is 0.68.

                                            The chart shows the harmonised index of consumer prices (HICP) of Romania, published by Eurostat with 2015 = 100, together with one-month-ahead forecasts from two random forests trained on the data up to December 2020 (the vertical dotted line). The first forest uses lagged levels of the index as features and forecasts the level. The second uses monthly log changes of the index and forecasts the next change, which is then added to the last observed level.

                                            The horizontal dotted line marks the largest level seen in training. After 2020 the price level rises steeply, and the chart shows what each forest does with values it has never seen.

                                            In brief

                                            • HICP (harmonised index of consumer prices) of Romania, Eurostat, 2015 = 100; one-month-ahead forecasts from random forests trained up to December 2020, on lagged levels or on monthly log changes
                                            TSA_ch9_regularisation_trees

                                            Interpreting the extrapolation failure

                                            Slide 37 in the PDF

                                            The highest index value in training was 111.3; by August 2026 the index had reached 173.4. The forest on levels stays near 110.8 for more than five years, because each of its forecasts is a mean of training levels and none of them is higher than 111.3. Its RMSE over the test period is 36.9 index points.

                                            The same forest applied to monthly changes, with the forecasts cumulated onto the last observed level, follows the index closely, with an RMSE of 0.71 index points. Monthly inflation rates after 2021 were higher than before, but they stayed within a range that the forest could learn, while the level moved far outside it. The lesson repeats Chapter 3: a non-stationary series is differenced or transformed before a tree sees it.

                                            In brief

                                            • The highest index in training was 111.3; by August 2026 the index reached 173.4
                                              • the forest on levels stays near 110.8 for five years: RMSE 36.9 index points
                                            • The same forest on monthly changes, then cumulated, follows the index: RMSE 0.71
                                            • Chapter 3 again: difference or transform non-stationary series before a tree sees them

                                            The features a model uses

                                            Slide 38 in the PDF

                                            A random forest or a boosted model has no coefficients to read, so its use of the features has to be measured. Permutation importance shuffles the values of one feature in the test data, which destroys its link with the target while keeping its distribution, and measures how much the forecast error grows. A feature that the model relies on produces a large increase; an unused feature produces none. The measure works for any model and is computed out of sample.

                                            Correlated features are a weakness of the method. If two features carry the same information, shuffling one of them leaves the other in place, so each appears less important than the pair really is. Trees also offer an impurity-based importance (mean decrease impurity, MDI), the total reduction of the squared error obtained by splits on each feature. It is computed on the training data and favours continuous features with many possible split points, so permutation importance is preferred.

                                            Importance describes the model, not the electricity system. A feature can be important because it is correlated with the true cause, and importance says nothing about what would happen if the feature changed.

                                            In brief

                                            • Permutation importance: shuffle one feature in the test data and measure how much the error grows
                                              • model-agnostic; measured out of sample; correlated features share (and hide) their importance
                                            • Impurity-based importance of trees (MDI, mean decrease impurity) is computed in training and favours continuous features: prefer permutation
                                            • Importance is not causality: it describes the model, not the electricity system

                                            Permutation importance of gradient boosting

                                            Slide 39 in the PDF
                                              Question for the room Move the horizons to 7 and 8 days. Which feature dominates at h = 8?
                                              Answer

                                              cos1 (+98 MW), followed by lag6 (+84 MW). At h = 7 lag0 still dominates (+317 MW), because at 7 days yt is the same weekday as the target.

                                              The chart shows permutation importances for two direct gradient boosting models, one for the load one day ahead and one for the load 14 days ahead. Both are trained on the data up to 31 December 2024 and evaluated on January to June 2025, and each feature is shuffled 20 times. The bars give the average increase in MAE, in megawatts (MW), for the eight most important features of each model.

                                              The names follow the feature table: lag0 is the load at the origin, lagk the load \(k\) days earlier, mean7 and mean28 rolling means, mon and sat weekday dummies, holiday the holiday flag, and cos1, cos2 the first two cosine Fourier terms of the annual cycle.

                                              In brief

                                              • Direct GB models for horizons 1 and 14 days; trained up to 31 December 2024, evaluated on January–June 2025; 20 random permutations per feature
                                              TSA_ch9_load_forecasting

                                              Interpreting the importances

                                              Slide 40 in the PDF

                                              One day ahead, two features dominate. Shuffling lag0 increases the MAE by 371 MW and shuffling lag6 by 117 MW. For the target \(y_{t+1}\), lag0 is yesterday's load and lag6 is \(y_{t-6}\), the load on the same weekday one week earlier: the persistence of the series and the weekly cycle.

                                              Fourteen days ahead the order changes: cos1 increases the MAE by 126 MW and lag0 by 89 MW. The recent values say less about a day two weeks away, and the annual cycle, carried by the Fourier term, gains weight. lag0 stays important because for \(h = 14\) the origin falls on the same weekday as the target, and lag7 is again a same-weekday value.

                                              In brief

                                              • One day ahead: lag0 (+371 MW of MAE when shuffled) and lag6 (+117 MW) dominate
                                                • yesterday and the same weekday last week: the persistence and the weekly cycle
                                              • Fourteen days ahead: cos1 (+126 MW) and lag0 (+89 MW)
                                                • the annual cycle (cos1, the Fourier term) gains weight as the recent values become less informative

                                              Recap: Trees and ensembles

                                              Slide 41 in the PDF

                                              A tree is a piecewise-constant function found by greedy splits, and it cannot extrapolate beyond the range of its training targets. A random forest averages decorrelated deep trees and lowers the variance; boosting adds small trees fitted to residuals and lowers the bias. Gradient boosting (LightGBM, XGBoost, HistGradientBoosting) is tuned through the learning rate, the number of trees and early stopping. Permutation importance measures, out of sample, which features a model uses.

                                              In brief

                                              • A tree is a piecewise-constant function found by greedy splits; it cannot extrapolate
                                              • Random forest averages decorrelated deep trees (less variance); boosting adds small trees on residuals (less bias)
                                              • GB (gradient boosting, LightGBM, XGBoost, HistGradientBoosting): learning rate, number of trees, early stopping
                                              • Permutation importance measures what the model uses, out of sample

                                              Section summary

                                              On the electricity load, ensembles of trees reduce the validation MSE of a single tree from 0.183 to about 0.10, against 0.303 for the seasonal naive forecast. Tree methods need stationary targets: on the price level, a forest trained on levels stays flat at the highest training value, while the same forest trained on monthly changes follows the index.

                                              Neural networks

                                              Neural networks are flexible function approximators built from layers of simple units. The section presents the multilayer perceptron, the way it is trained, and two networks designed for sequences: the recurrent network and the long short-term memory (LSTM) network. The evidence on when such networks help in forecasting closes the section.

                                              The multilayer perceptron

                                              Slide 42 in the PDF

                                              The multilayer perceptron (MLP) with one hidden layer of \(q\) neurons computes, for an input vector \(\mathbf{x}\),

                                              \[\begin{aligned} h_j &= g(b_j + \mathbf{w}_j'\mathbf{x}),\quad j = 1, \dots, q, \\ \hat y &= c + \sum_{j=1}^{q} v_j h_j . \end{aligned}\]
                                              (9.5)

                                              Each hidden neuron \(j\) forms a weighted sum of the inputs with weights \(\mathbf{w}_j\) and bias \(b_j\), and passes it through an activation function \(g\). The output is a linear combination of the hidden outputs \(h_j\), with weights \(v_j\) and constant \(c\). Common activations are ReLU, \(\max(0, z)\), the hyperbolic tangent and the logistic function. The activation is what makes the network nonlinear: with \(g(z) = z\) the whole network collapses to a linear regression.

                                              Hornik, Stinchcombe and White (1989) proved that such a network can approximate any continuous function on a bounded set as closely as desired, if it has enough hidden neurons. This universal approximation property explains the flexibility of networks; it does not say how many neurons are needed or how much data is required to estimate them. The number of parameters grows quickly: with 3 inputs and 5 hidden neurons there are \(3 \cdot 5 = 15\) input weights, 5 hidden biases, 5 output weights and 1 output constant, 26 parameters in all.

                                              The image shows the Mark I Perceptron, built at Cornell around 1960, the first trainable neural network, implemented in hardware.

                                              In brief

                                              The multilayer perceptron Mark I Perceptron, Cornell (1960): the first trainable neural network, built in hardware Image: J. C. Hay, A. E. Murray (1960), Mark I Perceptron operator's manual; public domain; Wikimedia Commons
                                              • MLP (multilayer perceptron), one hidden layer with \(q\) neurons:
                                                • \(h_j = g(b_j + \mathbf{w}_j'\mathbf{x})\), \(j = 1, \dots, q\);   \(\hat y = c + \sum_j v_jh_j\)
                                                • \(\mathbf{x}\): the inputs; \(\mathbf{w}_j\), \(b_j\): the weights and the bias of neuron \(j\); \(h_j\): its output; \(v_j\), \(c\): the output weights and constant
                                                • \(g\): activation function (ReLU \(\max(0, z)\), tanh, logistic); with \(g(z) = z\) the network is linear again
                                              • Universal approximation Hornik, Stinchcombe and White (1989): enough hidden neurons approximate any continuous function
                                                • parameters: 3 inputs and 5 neurons already give \(3 \cdot 5 + 5 + 5 + 1 = 26\) weights

                                              A small network and its activation functions

                                              Slide 43 in the PDF
                                                Question for the room How many parameters does the network with q = 10 hidden neurons have?
                                                Answer

                                                3·10 + 10 + 10 + 1 = 51. With 10 ReLU neurons the piecewise-linear function approximates tanh with maximum error 0.048, against 0.102 with 5.

                                                The left panel draws the network of the example for tomorrow's load: three inputs (today's load \(y_t\), the load on the same weekday of the previous week \(y_{t-6}\) and a holiday flag), a hidden layer of five neurons and one output, \(\hat y_{t+1}\). Every line is a weight; each hidden neuron receives all three inputs, and the output receives all five hidden values.

                                                The right panel plots the three activation functions. The logistic function rises from 0 to 1 and the hyperbolic tangent from \(-1\) to 1, both smoothly and both flat in the tails; ReLU is 0 for negative arguments and the identity for positive ones. The flat tails of the first two make their derivatives small for large inputs, which slows training in deep networks; ReLU avoids this for positive arguments, which is one reason for its popularity.

                                                In brief

                                                • Left: 3 inputs (yesterday, the same weekday last week, a holiday flag), 5 hidden neurons, one output; right: three activation functions
                                                TSA_ch9_neural_networks

                                                Training a network

                                                Slide 44 in the PDF

                                                A network is trained by minimising the training loss with gradient descent: the weights move in small steps against the gradient of the loss. The gradient with respect to every weight is computed by back-propagation (Rumelhart, Hinton and Williams, 1986), an application of the chain rule that passes the error from the output back through the layers. Stochastic gradient descent (SGD) and its variant Adam compute each step on a small random batch of rows instead of the whole sample; one pass through all the data is called an epoch.

                                                The loss of a network, as a function of its weights, has many local minima, so the result depends on the random starting weights. Two practices follow: fix the random seed so that the result can be reproduced, and average a few networks trained from different seeds, which forms an ensemble with a lower variance.

                                                Networks overfit easily and are regularised in three main ways: an L2 penalty on the weights, as in ridge regression; early stopping on validation data, as in boosting; and dropout, which switches off random neurons during training. Inputs and targets must be standardised, with the statistics of the training window. With a few thousand rows, as in the load data, small networks are appropriate; deep networks need many series or long high-frequency data.

                                                In brief

                                                • Minimise the training loss by gradient descent; the gradients come from back-propagation Rumelhart, Hinton and Williams (1986)
                                                  • SGD (stochastic gradient descent) and Adam use small random batches; one pass through the data is an epoch
                                                • The loss surface has many local minima: results depend on the random starting weights
                                                  • fix the seed, and average a few networks with different seeds (an ensemble)
                                                • Regularisation: an L2 penalty on the weights, early stopping, dropout (randomly switching off neurons)
                                                  • inputs and targets must be standardised with training-window statistics
                                                • Small data (a few thousand rows) favour small networks; deep networks need many series or long high-frequency data

                                                Recurrent networks and the LSTM (1/2): the recurrent network

                                                Slide 45 in the PDF

                                                A recurrent neural network (RNN) reads a sequence one step at a time and carries a hidden state that summarises the past. At each step the state is updated from the previous state and the new input:

                                                \[h_t = \tanh(W h_{t-1} + U x_t + b) ,\]

                                                where \(x_t\) is the input at time \(t\), \(h_t\) the hidden state, and \(W\), \(U\) and \(b\) are weights and a bias shared by all steps. The hyperbolic tangent keeps each component of the state between \(-1\) and 1. The forecast is read from the last state, for example \(\hat y_{t+1} = v'h_t + c\). Sharing the weights across steps is what lets the same network process sequences of any length.

                                                Training an RNN means differentiating through many steps, which multiplies many derivatives of the form \(\partial h_t/\partial h_{t-1}\). If they are smaller than 1, the product shrinks towards zero (vanishing gradients); if they are larger than 1, it explodes. In practice a plain RNN cannot learn dependencies that reach more than a few steps back.

                                                The review of Hewamalage, Bergmeir and Bandara (2021) summarises the evidence for forecasting. On single, short series, recurrent networks are rarely more accurate than ETS or ARIMA. They become competitive with many related series and enough data, as global models such as DeepAR (Salinas et al., 2020), which train one network on thousands of series.

                                                In brief

                                                • RNN (recurrent neural network): a hidden state carries the past forward, with the same weights at every step
                                                  • \(h_t = \tanh(Wh_{t-1} + Ux_t + b)\): the new state mixes the previous state and the new input
                                                  • \(x_t\): the input at \(t\); \(h_t\): the hidden state; \(W\), \(U\), \(b\): weights and bias, shared by all steps; \(\tanh\) keeps each component in \((-1, 1)\)
                                                  • the forecast is read from the last state, for example \(\hat y_{t+1} = v'h_t + c\)
                                                • The difficulty: training through many steps multiplies many derivatives
                                                  • the gradients vanish or explode: a plain RNN forgets what happened more than a few steps back
                                                • Honest summary Hewamalage, Bergmeir and Bandara (2021)
                                                  • on single, short series, recurrent networks rarely beat ETS or ARIMA
                                                  • they shine with many related series and enough data (global models, Salinas et al. (2020))

                                                Recurrent networks and the LSTM (2/2): the LSTM cell

                                                Slide 46 in the PDF

                                                The long short-term memory network of Hochreiter and Schmidhuber (1997) adds a cell state \(c_t\) that is updated additively and controlled by three gates. Each gate is a vector of numbers between 0 and 1, computed by a logistic layer from the previous output and the new input:

                                                \[\begin{aligned} f_t &= \sigma(W_f[h_{t-1}, x_t] + b_f), \\ i_t &= \sigma(W_i[h_{t-1}, x_t] + b_i), \\ o_t &= \sigma(W_o[h_{t-1}, x_t] + b_o), \\ \tilde c_t &= \tanh(W_c[h_{t-1}, x_t] + b_c). \end{aligned}\]

                                                Here \(\sigma(z) = 1/(1 + e^{-z})\) is the logistic function, \([h_{t-1}, x_t]\) stacks the two vectors, and \(W_f, W_i, W_o, W_c\) and \(b_f, b_i, b_o, b_c\) are weights and biases. The forget gate \(f_t\), the input gate \(i_t\) and the output gate \(o_t\) decide what is kept, what is added and what is passed on; \(\tilde c_t\) is the candidate new content.

                                                The cell and the output are then updated as

                                                \[\begin{aligned} c_t &= f_t \odot c_{t-1} + i_t \odot \tilde c_t, \\ h_t &= o_t \odot \tanh(c_t) , \end{aligned}\]
                                                (9.6)

                                                where \(\odot\) is the element-wise product. When the forget gate is close to 1, the old memory passes through many steps almost unchanged, and so does its gradient, which solves the vanishing-gradient problem of the plain RNN. The price is the number of weights: each gate is a full layer, and an LSTM with 32 units and 8 inputs has about 5,400 weights, four times as many as a plain RNN of the same size.

                                                In brief

                                                • LSTM (long short-term memory) Hochreiter and Schmidhuber (1997): a cell state \(c_t\) updated additively, controlled by three gates
                                                  • the gates: \(f_t = \sigma(W_f[h_{t-1}, x_t] + b_f)\) (forget), \(i_t = \sigma(W_i[h_{t-1}, x_t] + b_i)\) (input), \(o_t = \sigma(W_o[h_{t-1}, x_t] + b_o)\) (output)
                                                  • the candidate new content: \(\tilde c_t = \tanh(W_c[h_{t-1}, x_t] + b_c)\)
                                                  • \(\sigma(z) = 1/(1 + e^{-z}) \in (0, 1)\): the logistic function; \([h_{t-1}, x_t]\): the two vectors stacked; \(W_f, W_i, W_o, W_c\) and \(b_f, b_i, b_o, b_c\): weights and biases
                                                • The updates of the cell and of the output
                                                  • \(c_t = f_t \odot c_{t-1} + i_t \odot \tilde c_t\): with \(f_t\) close to 1 the old memory survives many steps; \(i_t\) sets how much new content enters
                                                  • \(h_t = o_t \odot \tanh(c_t)\): the output gate decides which part of the memory is passed on; \(\odot\): element-wise product
                                                • Each gate is a small logistic layer: an LSTM with 32 units and 8 inputs has about 5 400 weights

                                                A recurrent network and the LSTM cell

                                                Slide 47 in the PDF
                                                  Question for the room Choose the origin 6 April 2026. What is the MAE of the LSTM, against DHR?
                                                  Answer

                                                  661 MW against 344 MW. The LSTM sees no holiday flag and repeats the weekly pattern over Easter, although its training loss falls from 0.951 to 0.210.

                                                  The left diagram unrolls an RNN over three steps. Each green box is the hidden state at one time, each blue box an input, and the red arrows carry the state forward with the same weights at every step; the forecast \(\hat y_{t+1}\) is read from the last state.

                                                  The right diagram shows the LSTM cell. The red line is the cell state, which flows from \(c_{t-1}\) to \(c_t\) as a long memory; the three gates act on it from below. The forget gate scales the old memory, the input gate admits new content, and the output gate selects what part of the memory becomes the output \(h_t\). The equations under the diagram are those of (9.6).

                                                  In brief

                                                  • Left: the RNN unrolled over three steps; right: the LSTM cell state (red) and its three gates; \(\odot\): element-wise product
                                                  TSA_ch9_neural_networks

                                                  Sepp Hochreiter and the LSTM

                                                  Slide 48 in the PDF

                                                  In his 1991 diploma thesis in Munich, Sepp Hochreiter analysed why the gradients of recurrent networks vanish over long sequences. Together with Jürgen Schmidhuber he then proposed the LSTM (Hochreiter and Schmidhuber, 1997), whose additive cell state lets the gradient pass through many steps.

                                                  The paper became one of the most cited in neural computing. LSTMs powered speech recognition and machine translation systems for many years, until transformers, the architecture behind the foundation models of Chapter 11, replaced them in most of these tasks.

                                                  In brief

                                                  Sepp Hochreiter and the LSTM Sepp Hochreiter (2015), Johannes Kepler University Linz Photo: Eulenreich (2015); CC BY-SA 4.0; Wikimedia Commons
                                                  • In his 1991 diploma thesis in Munich, Hochreiter analysed why gradients vanish in recurrent networks
                                                  • With Jürgen Schmidhuber he proposed the LSTM Hochreiter and Schmidhuber (1997)
                                                    • one of the most cited papers in neural computing
                                                  • LSTMs powered speech recognition and machine translation before transformers (Chapter 11)

                                                  Recap: Neural networks

                                                  Slide 49 in the PDF

                                                  An MLP stacks layers of units \(g(b + \mathbf{w}'\mathbf{x})\) and is trained by gradient descent; the seed, the scaling of the data, the L2 penalty and early stopping decide whether training succeeds. RNNs and LSTMs process sequences, and the LSTM cell state avoids vanishing gradients. Small data favour small models; neural networks become competitive with many series or long data.

                                                  In brief

                                                  • MLP: layers of \(g(b + \mathbf{w}'\mathbf{x})\); trained by gradient descent; seed, scaling, L2 penalty, early stopping
                                                  • RNN and LSTM process sequences; the LSTM cell state avoids vanishing gradients
                                                  • Small data favour small models; neural networks need many series or long data

                                                  Section summary

                                                  Neural networks can approximate any continuous function, but this flexibility has to be paid for with data and careful training. For a single daily series with a few thousand observations, a small MLP is a reasonable candidate; recurrent networks and the LSTM are designed for long sequences and many related series, and the next section shows what happens when they are applied to one series of moderate length.

                                                  Application: electricity load against Chapter 4

                                                  This section puts the ML methods into the forecasting exercise of Chapter 4. The statistical models of that chapter and six ML models forecast the daily electricity load of Romania from the same 26 origins, for the same 14 horizons, and are compared with the same accuracy measures and the Diebold–Mariano test.

                                                  The forecasting exercise

                                                  Slide 50 in the PDF

                                                  The evaluation design is that of Chapter 4. There are 26 forecast origins, one every 14 days from 30 June 2025 to 15 June 2026, and from each origin every model forecasts the next 1 to 14 days. The training window is expanding: at each origin it contains all data from 1 January 2022. The statistical models are those of Chapter 4: the seasonal naive forecast, ETS, seasonal ARIMA (SARIMA), dynamic harmonic regression (DHR) and the combination of these forecasts.

                                                  The ML models are refitted at every origin. Ridge, a random forest of 200 trees and gradient boosting with 300 trees and learning rate \(\nu = 0.05\) use the direct strategy, with one model per horizon. A second gradient boosting model uses the recursive strategy on the one-step table. Two neural networks use the multi-output strategy: an MLP, averaged over 3 networks with 32 hidden neurons, and an LSTM that reads the last 56 days and outputs the next 14.

                                                  The MASE divides the MAE of the forecasts by a fixed scale, the in-sample MAE of the weekly seasonal naive method, 309 MW. A MASE below 1 means that the model is more accurate out of sample than the seasonal naive forecast is in sample.

                                                  In brief

                                                  The forecasting exercise 750 kV line at Mărășești: the Romanian transmission grid Photo: TrainSimFan (2022); CC BY-SA 4.0; Wikimedia Commons
                                                  • The evaluation design of Chapter 4: 26 forecast origins every 14 days, 30 June 2025 to 15 June 2026
                                                    • horizon 1–14 days; expanding window from 1 January 2022
                                                    • statistical models (Chapter 4): seasonal naive, ETS, SARIMA, DHR (dynamic harmonic regression), their combination
                                                  • ML models, refitted at each origin:
                                                    • direct: ridge, random forest (200 trees), GB (300 trees, \(\nu = 0.05\)), one model per horizon
                                                    • recursive: GB on the one-step table; MIMO: an MLP (3 networks, 32 neurons) and an LSTM (56 days in, 14 out)
                                                  • MASE scale: in-sample MAE of the weekly seasonal naive method, 309 MW

                                                  Results: 26 origins, 14 horizons

                                                  Slide 51 in the PDF

                                                  The table gives, for each model, the MAE and the RMSE in GW over all 364 forecasts (26 origins times 14 horizons), the MASE, and the p-value of the Diebold–Mariano test against DHR, the most accurate model. The test is applied to the 26 origin-average absolute errors, with the small-sample correction of Harvey, Leybourne and Newbold (1997). A small p-value means that the difference in accuracy from DHR is unlikely to be due to chance.

                                                  The table splits into three groups. DHR has the lowest MAE, 0.276 GW, and ridge is practically equal, 0.277 GW, with a slightly lower RMSE, 0.337 against 0.350 GW. The MLP (MASE 0.95), direct gradient boosting (0.97), the combination (0.99), recursive gradient boosting (0.99) and the random forest (1.00) form a middle group whose differences from DHR are not significant at 5%. The seasonal naive forecast, ETS, SARIMA and the LSTM are significantly less accurate than DHR, with p-values from below 0.001 to 0.016.

                                                  In brief

                                                  ModelMAE (GW)RMSE (GW)MASEDM p, against DHR
                                                  Seasonal naive0.3960.5291.28\(<\) 0.001
                                                  ETS0.3810.4921.230.015
                                                  SARIMA0.3840.4771.240.003
                                                  DHR0.2760.3500.89–
                                                  Combination0.3050.3800.990.065
                                                  Ridge0.2770.3370.900.942
                                                  RF0.3080.3881.000.068
                                                  GB direct0.2980.3690.970.163
                                                  GB recursive0.3070.3880.990.113
                                                  MLP0.2940.3620.950.546
                                                  LSTM0.3570.4471.160.016

                                                  MASE and accuracy by horizon

                                                  Slide 52 in the PDF

                                                  Worked example

                                                  \[\text{MASE}=\frac{\text{MAE}}{\frac{1}{n-m}\sum_{t=m+1}^{n}|y_t-y_{t-m}|}\]
                                                      Question for the room Change the period of the scale to m = 1. What do the scale and the MASE of DHR become?
                                                      Answer

                                                      The scale becomes 371 MW and the MASE 0.744 (309 MW and 0.894 with m = 7). Day-to-day differences are larger than week-to-week ones, because the step into the weekend changes the load; the scale moves the level, not the ranking of the models.

                                                        Question for the room Drop the first 10 origins. Does the difference between ridge and DHR stay not significant?
                                                        Answer

                                                        Yes: on 16 origins ridge has MASE 0.92 and DHR 0.93, and DM p = 0.791. Not significant means that the test does not separate the difference from noise, not that the models are equal.

                                                        Worked example

                                                        \[d_t=\overline{|e|}_{t}^{\,A}-\overline{|e|}_{t}^{\,B},\quad \hat V=\frac{1}{n}\Big(\hat\gamma_0+2\sum_{k=1}^{h-1}\hat\gamma_k\Big),\quad DM=\frac{\bar d}{\sqrt{\hat V}},\quad DM^{*}=c_h\,DM\sim t_{n-1}\]
                                                            Question for the room Include 2 autocovariances (h = 3) for ridge against DHR. What does p become?
                                                            Answer

                                                            0.953 (DM* = 0.060), against 0.942 without autocovariances: d̄ = 0.9 MW is small relative to its standard error, however the variance is estimated.

                                                              Question for the room Use only the first 10 origins (up to 3 November 2025). Which model has the lowest MASE?
                                                              Answer

                                                              Still DHR (0.83), followed by ridge (0.86). The LSTM has 0.95 here, but over all 26 origins it rises to 1.16.

                                                              The left panel ranks the eleven models by MASE: blue bars are the models of Chapter 4, red bars the ML models, and the dotted line marks 1. DHR, ridge and the MLP are furthest below the line; direct and recursive gradient boosting and the combination are just below it, the random forest is at 1.00, and the LSTM, SARIMA, ETS and the seasonal naive forecast are above it.

                                                              The right panel shows the MAE by horizon for six models. Errors grow with the horizon for every model, but not smoothly: most models have their largest errors at horizons 10, 11 and 14. DHR is the most accurate model at short horizons, with an MAE of 0.18 GW one day ahead. Direct gradient boosting stays close to DHR from horizon 3 onwards, while the seasonal naive forecast and the LSTM show the largest peaks.

                                                              In brief

                                                              • Left: MASE of the eleven models (blue: Chapter 4; red: ML); right: MAE by horizon for six models
                                                              TSA_ch9_load_forecasting

                                                              Interpreting the load results

                                                              Slide 53 in the PDF

                                                              DHR, with a MASE of 0.89, and ridge, with 0.90, are the most accurate models, and the difference between them is not significant (Diebold–Mariano p-value 0.942). Both are linear regressions with the right calendar features: Fourier terms, weekday effects and holiday dummies. The trees and the networks do not improve on them.

                                                              Direct gradient boosting (MASE 0.97), the MLP (0.95) and the random forest (1.00) are clearly more accurate than the seasonal naive forecast, with Diebold–Mariano p-values of 0.0015, 0.0074 and 0.0015, and they have a lower MASE than SARIMA and ETS, a difference not tested with Diebold–Mariano. The direct and recursive versions of gradient boosting are close, 0.97 and 0.99, so the strategy matters less than the features here. The LSTM, with a MASE of 1.16, is less accurate than every other model except the seasonal naive forecast, SARIMA and ETS.

                                                              The lesson of the exercise is that the features matter more than the algorithm. With four years of daily data on a single series, a well-specified linear model that knows the calendar is hard to beat, and the flexibility of trees and networks has little to add.

                                                              In brief

                                                              • DHR (MASE 0.89) and ridge (0.90) are the most accurate; the difference is not significant (p = 0.942)
                                                                • both are linear models with the right calendar features; the trees and networks do not beat them
                                                              • GB direct (0.97), MLP (0.95), random forest (1.00) beat the seasonal naive method clearly (DM p = 0.0015; 0.0074; 0.0015) and have a lower MASE than SARIMA and ETS (not tested with DM)
                                                                • direct and recursive GB are close (0.97 and 0.99); the LSTM (1.16) is worse than every model except the naive, SARIMA and ETS
                                                              • Lesson: features matter more than the algorithm; with four years of daily data, a well-specified linear model is hard to beat

                                                              Forecasts around Orthodox Easter 2026

                                                              Slide 54 in the PDF
                                                                Question for the room Choose the origin 20 April 2026, after Easter. Which model has the lowest MAE?
                                                                Answer

                                                                The MLP, with 255 MW; GB recursive has 260, GB direct 276, the seasonal naive 283, DHR 286 and the LSTM 398 MW. Without Easter in the window the differences between the models are small.

                                                                The chart zooms in on one origin, 6 April 2026, a week before Orthodox Easter on 12 April. The black line is the actual load, and the coloured lines are the 14-day forecasts of five models from the vertical dotted line onwards. The actual load falls to about 4 GW on Easter Sunday and stays below its usual level during the following week.

                                                                The MAE in this window measures how each model handled the holiday: 344 MW for DHR, 536 MW for direct gradient boosting, 503 MW for recursive gradient boosting and 661 MW for the LSTM. DHR forecasts the deepest dip on Easter Sunday; the tree models forecast a shallower one, and the LSTM and the seasonal naive forecast repeat the ordinary weekly pattern, with only the usual Sunday dip.

                                                                In brief

                                                                • Origin 6 April 2026; 14-day forecasts; MAE in this window: DHR 344 MW, GB direct 536 MW, GB recursive 503 MW, LSTM 661 MW
                                                                TSA_ch9_load_forecasting

                                                                Interpreting the Easter window

                                                                Slide 55 in the PDF

                                                                All models except the LSTM see the Easter flag among their features, but the training data contain only four Orthodox Easters, from 2022 to 2025. A tree can isolate the Easter days only by splitting on the flag, and with four episodes such splits are rare and their leaves are mixed with ordinary days, so the forecast dip is too shallow. DHR estimates one Easter coefficient from the same four episodes; the regression structure pools them efficiently and gives a dip of the right depth.

                                                                The LSTM receives no holiday flag. It repeats the weekly pattern of the recent past and misses the holiday entirely. Rare events therefore need either structure, such as a regression coefficient for the event, or many series that share the event, which a global model can learn from.

                                                                In brief

                                                                • All models except the LSTM see the Easter flag, but the trees have only four Easters in training: they predict a dip that is too shallow
                                                                  • DHR estimates one Easter coefficient from the same four episodes and pools the information more efficiently
                                                                • The LSTM sees no holiday flag: it repeats the weekly pattern and misses the holiday entirely
                                                                • Rare events need either structure (a regression coefficient) or many series that share the event (a global model)

                                                                Recap: Electricity load

                                                                Slide 56 in the PDF

                                                                With the same origins, the same horizons and the same error measures, the ML models enter the comparison of Chapter 4 on equal terms. Ridge and DHR do not differ significantly (p = 0.942); gradient boosting, the random forest and the MLP have a lower MASE than SARIMA and ETS; the LSTM is the weakest of the ML models. Calendar features and holidays decide the ranking.

                                                                In brief

                                                                • Same origins, same horizon, same errors: ML models enter the comparison of Chapter 4
                                                                • Ridge and DHR: no significant difference (p = 0.942); GB, random forest and the MLP have a lower MASE than SARIMA and ETS; the LSTM is the weakest ML model
                                                                • Calendar features and holidays decide the ranking

                                                                Section summary

                                                                On the daily electricity load of Romania, the most accurate forecasts come from linear models with calendar features: DHR (MASE 0.89) and ridge (0.90), whose difference is not significant. Tree ensembles and the MLP are more accurate than the seasonal naive forecast, SARIMA and ETS, but not than DHR, and the LSTM, without a holiday flag, ranks near the bottom. Around Easter, a regression coefficient estimated from four episodes beats trees that see the same flag.

                                                                Prediction intervals

                                                                A point forecast says nothing about its uncertainty, yet many decisions depend on it. The section presents two ways of building prediction intervals for ML models: quantile regression with the pinball loss, which estimates the quantiles directly, and split conformal prediction, which widens a point forecast by the empirical quantile of out-of-sample errors. Both are applied to the load forecasts of the previous section.

                                                                Quantile regression and the pinball loss

                                                                Slide 57 in the PDF

                                                                A grid operator needs more than the expected load: it needs the load that will not be exceeded with 95% probability, to plan reserves. The \(\tau\)-quantile \(q_\tau\) of \(y_{t+h}\) is the value with \(P(y_{t+h} \le q_\tau) = \tau\), and a 90% prediction interval is the pair \([q_{0.05}, q_{0.95}]\). A model that forecasts quantiles therefore gives intervals directly.

                                                                Quantiles are estimated by minimising the pinball loss of Koenker and Bassett (1978):

                                                                \[L_\tau(y, q) = \begin{cases} \tau\,(y - q), & y \ge q, \\ (1 - \tau)(q - y), & y < q. \end{cases}\]
                                                                (9.7)

                                                                The loss is asymmetric. For \(\tau = 0.95\), an outcome above the forecast costs 0.95 per unit and an outcome below costs 0.05 per unit, so the forecast is pushed up until only 5% of the outcomes exceed it. Its expected value is minimised exactly by the true \(\tau\)-quantile, and for \(\tau = 0.5\) it equals half the absolute error, whose minimiser is the median.

                                                                An example with \(\tau = 0.95\) and a forecast \(q = 7.0\) GW: an outcome of 7.4 GW costs \(0.95 \cdot 0.4 = 0.380\), while an outcome of 6.5 GW costs only \(0.05 \cdot 0.5 = 0.025\). Any learner that can minimise a custom loss can forecast quantiles: linear quantile regression, gradient boosting with loss='quantile', or a neural network.

                                                                In brief

                                                                • A point forecast is not enough: a grid operator needs the load it will not exceed with 95% probability
                                                                  • the \(\tau\)-quantile \(q_\tau\) of \(y_{t+h}\): \(P(y_{t+h} \le q_\tau) = \tau\); a 90% interval: \([q_{0.05}, q_{0.95}]\)
                                                                • Pinball loss: \(L_\tau(y, q) = \tau(y - q)\) if \(y \ge q\), and \((1 - \tau)(q - y)\) if \(y < q\) Koenker and Bassett (1978)
                                                                  • its expected value is minimised by the true quantile; with \(\tau = 0.5\) it is half the absolute error
                                                                  • example, \(\tau = 0.95\), \(q = 7.0\) GW: \(y = 7.4\) costs \(0.95 \cdot 0.4 = 0.380\); \(y = 6.5\) costs \(0.05 \cdot 0.5 = 0.025\)
                                                                • Any learner can minimise the pinball loss: linear quantile regression, quantile GB (loss='quantile'), networks

                                                                Split conformal prediction

                                                                Slide 58 in the PDF

                                                                Conformal prediction, in the split form of Lei et al. (2018), builds an interval around any point forecast from the errors of the model on data it has not seen. The procedure has three steps. First, the model is fitted on the training data without the last \(n\) points, which form the calibration set. Second, the absolute errors \(|e_i|\) of the model on the calibration set are computed and sorted. Third, the interval for a new forecast \(\hat y\) is

                                                                \[\hat y \pm \hat q, \qquad \hat q = |e|_{(\lceil (n+1)(1-\alpha) \rceil)} ,\]
                                                                (9.8)

                                                                where \(|e|_{(k)}\) is the \(k\)-th smallest absolute calibration error, \(\alpha\) the target miscoverage (\(\alpha = 0.1\) for a 90% interval) and \(\lceil\cdot\rceil\) rounding up.

                                                                The guarantee is that the interval covers the outcome with probability at least \(1 - \alpha\) if the errors are exchangeable, that is, if their joint distribution does not depend on their order. Time-series errors are not exchangeable: their size changes with the season and they are correlated. The guarantee is therefore only approximate, and the remedy is to calibrate on recent data and to check the coverage walk-forward.

                                                                In the load application the calibration set has \(n = 182\) days and \(\alpha = 0.1\), so \(\lceil 183 \cdot 0.9 \rceil = 165\) and \(\hat q\) is the 165-th smallest of the 182 absolute errors. A separate calibration set is used for each horizon and each origin, so the width of the interval grows with the horizon.

                                                                In brief

                                                                • Idea Lei et al. (2018): use the errors of the model on data it has not seen to set the width of the interval
                                                                  • 1. fit the model on the training data except the last \(n\) points (the calibration set)
                                                                  • 2. compute the absolute errors \(|e_i|\) on the calibration set and sort them
                                                                  • 3. \(\hat q\) = the \(\lceil (n+1)(1 - \alpha) \rceil\)-th smallest; interval \(\hat y \pm \hat q\)
                                                                  • \(\alpha\): the target miscoverage (\(\alpha = 0.1\) for a 90% interval); \(\lceil\cdot\rceil\): rounding up
                                                                • Guarantee: coverage at least \(1 - \alpha\) if the errors are exchangeable (their order does not matter)
                                                                  • time series are not exchangeable: the guarantee is approximate; use recent calibration data and recheck the coverage walk-forward
                                                                • Here: \(n = 182\) days, \(\alpha = 0.1\), \(\hat q\) = the 165-th smallest absolute error, one calibration set per horizon and origin

                                                                90% intervals for the load

                                                                Slide 59 in the PDF

                                                                Worked example

                                                                \[L_\tau(y,q)=\begin{cases}\tau\,(y-q), & y\ge q\\(1-\tau)\,(q-y), & y
                                                                    Question for the room Set τ = 0.5. What do y = 7.4 and y = 6.5 cost for q = 7.0?
                                                                    Answer

                                                                    0.200 and 0.250: with τ = 0.5 the loss is half the absolute error (0.4/2 and 0.5/2), and the minimum of the mean loss moves to the median of the 364 values.

                                                                    Worked example

                                                                    \[k=\lceil (n+1)(1-\alpha)\rceil,\qquad \hat q=|e|_{(k)},\qquad \hat y\pm\hat q\]
                                                                        Question for the room Lower the level to 50%. What is k and what coverage does the interval reach?
                                                                        Answer

                                                                        k = ⌈183·0.5⌉ = 92 and q̂ = 175 MW for h = 1; the coverage over the 364 forecasts is 49.5%, with mean width 509 MW.

                                                                          Question for the room Lower the conformal level to 80%. What coverage does it reach?
                                                                          Answer

                                                                          77.5%, with mean width 1000 MW: 13.7% of the outcomes fall below the interval and 8.8% above. Quantile GB stays at 66.5% for a nominal 90%.

                                                                          The chart evaluates the two methods on the 364 direct gradient boosting forecasts of Section 6, all origins and all horizons. The left panel shows the outcomes (black dots), the 90% intervals of quantile gradient boosting (shaded band) and those of split conformal prediction (dashed lines). The right panel shows the empirical coverage, the share of outcomes inside the interval, by horizon; the dotted line marks the nominal 90%.

                                                                          The shaded band is visibly narrower than the conformal interval, and many outcomes, especially the low ones on Sundays and holidays, fall outside it. The conformal coverage stays close to 90% at most horizons, while the coverage of quantile boosting stays between about 55% and 75%.

                                                                          In brief

                                                                          • Direct GB, the 364 forecasts of Section 6 (all origins and horizons); left: intervals and outcomes; right: coverage by horizon
                                                                          TSA_ch9_load_forecasting

                                                                          Interpreting the intervals

                                                                          Slide 60 in the PDF

                                                                          Quantile gradient boosting covers 66.5% of the outcomes, with a mean width of 884 MW; split conformal prediction covers 90.4%, with a mean width of 1,310 MW. For quantile boosting, 19.0% of the outcomes fall below the interval and 14.6% above it, against a nominal 5% on each side. Its intervals are too narrow on both sides, and more so on the lower side, where the holiday drops are.

                                                                          Quantiles estimated by trees are noisy in the tails, where each leaf contains few extreme observations, and they reflect the training fit, which is tighter than the out-of-sample errors. Conformal calibration uses the real out-of-sample errors of the point forecast to set the width, so it corrects this optimism. An interval is only informative together with its empirical coverage, which should always be reported.

                                                                          In brief

                                                                          • Quantile GB covers 66.5% with mean width 884 MW; split conformal covers 90.4% with width 1310 MW
                                                                            • quantile GB: 19.0% of outcomes below and 14.6% above the interval (nominal 5% and 5%)
                                                                          • Quantiles estimated by trees are noisy in the tails and too narrow; conformal calibration corrects the width with real out-of-sample errors
                                                                          • Always report the empirical coverage next to an interval

                                                                          Recap: Prediction intervals

                                                                          Slide 61 in the PDF

                                                                          Quantile forecasts minimise the pinball loss, and an interval is a pair of quantiles. Split conformal prediction builds the interval \(\hat y \pm \hat q\) from an empirical quantile of calibration errors. In both cases the coverage is checked walk-forward.

                                                                          In brief

                                                                          • Quantile forecasts minimise the pinball loss; an interval is a pair of quantiles
                                                                          • Split conformal: \(\hat y \pm\) an empirical quantile of calibration errors
                                                                          • Check the coverage walk-forward

                                                                          Section summary

                                                                          On the load forecasts, nominal 90% intervals from quantile gradient boosting cover only 66.5% of the outcomes, while split conformal intervals, about 50% wider, cover 90.4%. The coverage of an interval has to be measured out of sample, and calibration on real forecast errors is a simple way to obtain the coverage that was promised.

                                                                          Local and global models: Romanian inflation

                                                                          All models so far were local: one model per series, estimated on that series alone. A global model is estimated on the pooled data of many related series. The section explains the distinction and tests it on Romanian inflation, with gradient boosting trained either on Romania alone or on the 27 countries of the European Union (EU).

                                                                          Local and global models

                                                                          Slide 62 in the PDF

                                                                          A local model is estimated on one series only, as ARIMA, ETS and every model of the previous sections. The number of observations of that series limits the complexity the model can afford: a monthly series of 20 years has 240 observations, too few for a large ensemble of trees or a neural network.

                                                                          A global model is estimated on the pooled rows of many related series; Montero-Manso and Hyndman (2021) and Januschowski et al. (2020) discuss this class of models. Every series contributes rows with the same features, such as scaled lags and calendar variables, and an identifier of the series can be added. The model has many more rows per parameter, so complex learners such as gradient boosting and neural networks become usable. The winners of the M4 and M5 competitions were global models.

                                                                          The risk is the reverse of the benefit. The pooled series must share their dynamics; if they do not, a global model averages away what is special about each one.

                                                                          In brief

                                                                          • Local model: one model per series, estimated on that series only (ARIMA, ETS, the models so far)
                                                                            • few observations per series limit the complexity a local model can afford
                                                                          • Global model: one model estimated on the pooled rows of many related series Montero-Manso and Hyndman (2021), Januschowski et al. (2020)
                                                                            • the same features for every series (scaled lags, calendar); optionally a series identifier
                                                                            • more data per parameter: complex learners (GB, networks) become usable; the winners of M4 and M5 were global
                                                                          • Risk: the series must share dynamics; a global model may average away what is special about one series

                                                                          The inflation exercise

                                                                          Slide 63 in the PDF

                                                                          The target is the 12-month inflation rate of Romania measured by the HICP, \(\pi_t = 100(P_t/P_{t-12} - 1)\), where \(P_t\) is the index in month \(t\), forecast at horizons of 1, 3, 6 and 12 months. The strategy is direct, and every model forecasts the change \(\pi_{t+h} - \pi_t\) rather than the level, so that the trees never have to extrapolate.

                                                                          The features at the origin \(t\) are the inflation rates \(\pi_t, \pi_{t-1}, \pi_{t-2}, \pi_{t-3}, \pi_{t-6}, \pi_{t-12}\), the last three monthly log changes of the index with their 3- and 6-month sums, and the calendar month. Six models are compared: the random walk (RW), which forecasts no change, \(\hat\pi_{t+h} = \pi_t\); an AR model estimated by OLS on the same features; the lasso; a random forest; local gradient boosting, trained on Romania only; and global gradient boosting, trained on the pooled data of the 27 EU countries.

                                                                          The validation is walk-forward with yearly refits: every January, each model is re-estimated on all rows from 2006 whose target month is already observed, so that no training target lies after the origin. The test origins run from January 2016 to July 2026. Medeiros et al. (2021) found that random forests forecast US inflation better than standard benchmarks when they use more than 100 predictors; the exercise asks whether a similar gain appears with the few predictors available for Romania.

                                                                          In brief

                                                                          • Target: 12-month HICP inflation of Romania (Eurostat), \(\pi_t = 100(P_t/P_{t-12} - 1)\), \(P_t\): the HICP index, at horizons \(h = 1, 3, 6, 12\) months
                                                                            • direct strategy; the models forecast the change \(\pi_{t+h} - \pi_t\), so trees never extrapolate the level
                                                                          • Features at \(t\): \(\pi_t, \pi_{t-1}, \pi_{t-2}, \pi_{t-3}, \pi_{t-6}, \pi_{t-12}\), the last three monthly log changes and their 3- and 6-month sums, the calendar month
                                                                            • models: random walk (RW, \(\hat\pi_{t+h} = \pi_t\)), AR (OLS on the same features), lasso, random forest, GB local (Romania only), GB global (27 EU countries pooled)
                                                                          • Walk-forward: refit every January on all rows whose target month is already observed (from 2006); test origins January 2016–July 2026
                                                                          • Related evidence: Medeiros et al. (2021) find that random forests beat the benchmarks for US inflation with more than 100 predictors

                                                                          Romanian inflation: forecasts and relative RMSE

                                                                          Slide 64 in the PDF
                                                                            Question for the room Choose the origins of 2021–2023. What is the relative RMSE of the AR at h = 6?
                                                                            Answer

                                                                            0.83 (0.88, 0.86, 0.83 and 0.98 at h = 1, 3, 6, 12): in the high-inflation years the AR stays below the random walk up to 6 months, while GB global has 1.10 at h = 6.

                                                                              Question for the room Start the test origins in 2022. What is the relative RMSE of GB local at 12 months?
                                                                              Answer

                                                                              0.64 (1.21 over all origins), while GB global has 1.19. On the few origins of the disinflation after the 2022 peak the 12-month ranking reverses: a comparison on a few dozen errors is fragile.

                                                                              The left panel shows the 12-month inflation rate of Romania (black) and the 12-month-ahead forecasts of four models, each plotted at its target month, so that a forecast and its outcome are vertically aligned. Inflation rose from below zero in 2016 to a peak in late 2022 and then declined; the forecasts made a year earlier follow it with a lag and miss the 2022 surge.

                                                                              The right panel gives, for each model and horizon, the RMSE divided by the RMSE of the random walk. A value below the dotted line at 1 means that the model is more accurate than forecasting no change. The AR model lies clearly below 1 up to 6 months, while most ML models lie at or above it.

                                                                              In brief

                                                                              • Left: inflation and the 12-month-ahead forecasts plotted at the target month; right: RMSE relative to the random walk
                                                                              TSA_ch9_inflation_global

                                                                              Inflation results

                                                                              Slide 65 in the PDF

                                                                              The table gives the RMSE in percentage points (pp) for each model and horizon, with the ratio to the random walk in parentheses. The random walk has an RMSE of 0.63 pp at 1 month, 1.41 pp at 3 months, 2.25 pp at 6 months and 3.90 pp at 12 months: errors grow quickly with the horizon. The AR model has ratios of 0.90 at 1, 3 and 6 months and 1.10 at 12 months. The lasso and the random forest have ratios between 1.01 and 1.09, and gradient boosting is worse at most horizons, with ratios of up to 1.21 for the local model at 12 months.

                                                                              The Diebold–Mariano test, based on squared errors and with \(h - 1\) autocovariances to account for the overlap of \(h\)-step errors, gives p-values for AR against the random walk of 0.019 at 1 month, 0.186 at 3 months and 0.388 at 6 months. For global against local gradient boosting the p-values are 0.117, 0.113, 0.999 and 0.828 at the four horizons. The number of test origins falls from 127 months at \(h = 1\) to 116 months at \(h = 12\).

                                                                              In brief

                                                                              RMSE (pp), relative to RW\(h = 1\)\(h = 3\)\(h = 6\)\(h = 12\)
                                                                              RW0.63 (1.00)1.41 (1.00)2.25 (1.00)3.90 (1.00)
                                                                              AR0.57 (0.90)1.28 (0.90)2.03 (0.90)4.28 (1.10)
                                                                              Lasso0.64 (1.01)1.45 (1.02)2.26 (1.01)4.10 (1.05)
                                                                              RF0.64 (1.02)1.49 (1.06)2.33 (1.04)4.25 (1.09)
                                                                              GB local0.70 (1.11)1.64 (1.16)2.37 (1.05)4.72 (1.21)
                                                                              GB global0.64 (1.01)1.37 (0.97)2.37 (1.05)4.54 (1.17)
                                                                              • DM p against RW, AR: 0.019 (\(h = 1\)), 0.186 (\(h = 3\)), 0.388 (\(h = 6\)); GB global against GB local: 0.117, 0.113, 0.999, 0.828
                                                                              • Test origins: 127 (\(h = 1\)) to 116 (\(h = 12\)) months; DM with squared errors and \(h - 1\) autocovariances

                                                                              Interpreting the inflation results

                                                                              Slide 66 in the PDF

                                                                              The linear AR model is the most accurate up to 6 months, with an RMSE about 10% below the random walk, but its advantage is significant only at 1 month (p = 0.019). No ML model beats the random walk significantly at any horizon. At 12 months no model beats the random walk at all: the 2022 peak, 14.6% in November 2022, came from energy prices, an external shock that the past of inflation did not contain.

                                                                              The global gradient boosting model is more accurate than the local one at short horizons, with an RMSE of 1.37 against 1.64 pp at 3 months, but the difference is not significant (p = 0.113). Pooling 27 countries gives the trees enough rows to learn from; the local model, with 120 to 240 rows at each refit, overfits.

                                                                              The sample size limits what can be concluded. About 120 monthly test errors, strongly affected by one episode, cannot separate models whose RMSEs differ by 5%. The honest conclusion for most comparisons is that the data give no evidence of a difference, which is not the same as evidence that the models are equally accurate.

                                                                              In brief

                                                                              • The linear AR is the best model up to 6 months (about 10% below the random walk); no ML model beats the random walk significantly
                                                                                • at 12 months nothing beats the random walk: the 2022 peak (14.6% in November 2022) came from energy prices, not from the past of inflation
                                                                              • The global GB is better than the local one at short horizons (RMSE 1.37 against 1.64 at \(h = 3\)), but not significantly (p = 0.113)
                                                                                • pooling 27 countries gives the trees enough rows; the local GB, with 120–240 rows per refit, overfits
                                                                              • About 120 monthly test errors cannot separate models whose RMSEs differ by 5%: the honest conclusion is “no evidence”

                                                                              Recap: Local and global models

                                                                              Slide 67 in the PDF

                                                                              A global model pools many related series and lets flexible learners use more data. For Romanian inflation, the AR model is the most accurate, global gradient boosting is more accurate than local gradient boosting, and no model beats the random walk at 12 months. With small test samples, Diebold–Mariano p-values must be reported next to RMSE rankings.

                                                                              In brief

                                                                              • A global model pools many related series and lets flexible learners use more data
                                                                              • Romanian inflation: AR is best; global GB beats local GB; nothing beats the random walk at 12 months
                                                                              • Small samples: report DM p-values, not only RMSE rankings

                                                                              Section summary

                                                                              Monthly inflation offers too few observations per country for flexible learners: a linear AR is the most accurate model up to 6 months, and gradient boosting gains from pooling 27 countries without becoming better than the random walk. The comparison also shows the limits of inference with about 120 monthly test errors.

                                                                              Applications in finance: volatility and the sign of returns

                                                                              Chapters 1 and 5 showed that the size of daily returns is predictable and their direction is not. The section tests whether machine learning changes this picture. The first application forecasts realised volatility against the HAR model; the second predicts the sign of tomorrow's return against the correct baseline.

                                                                              Realised volatility: HAR against ML (1/2)

                                                                              Slide 68 in the PDF

                                                                              The target is the logarithm of the mean daily variance over the next 5 trading days. The daily variance is not observed, so it is estimated from the open, high, low and close prices of each day, following Garman and Klass (1980):

                                                                              \[\begin{aligned} \hat\sigma_t^2 &= [\ln(O_t/C_{t-1})]^2 + \tfrac12[\ln(H_t/L_t)]^2 \\ &\quad - (2\ln 2 - 1)[\ln(C_t/O_t)]^2 , \end{aligned}\]

                                                                              where \(O_t\), \(H_t\), \(L_t\) and \(C_t\) are the open, high, low and close of day \(t\). The first term is the squared overnight return; the other two form the Garman–Klass estimator of the variance during the trading day, which uses the daily range and is much more precise than a squared close-to-close return. Volatility is persistent and predictable (Chapter 5), so this is a problem where ML could find structure.

                                                                              The benchmark is the heterogeneous autoregressive (HAR) model of Corsi (2009), an OLS regression on the log variance over the last day, week and month:

                                                                              \[\begin{aligned} \ln \bar\sigma^2_{t+1:t+5} &= c + \beta_d \ln\hat\sigma_t^2 + \beta_w \ln\bar\sigma^2_{t-4:t} \\ &\quad + \beta_m \ln\bar\sigma^2_{t-21:t} + u_t , \end{aligned}\]
                                                                              (9.9)

                                                                              where \(\bar\sigma^2_{a:b}\) is the mean of \(\hat\sigma^2\) from day \(a\) to day \(b\) and \(u_t\) is the error. The three averages represent traders with daily, weekly and monthly horizons, and together they reproduce the slow decay of volatility. The ML models receive the three HAR features and 8 more: further lags of the daily variance, the quarterly mean, the daily return, its negative part and its absolute value.

                                                                              In brief

                                                                              • Target: the log of the mean daily variance over the next 5 days; daily variance proxy from open, high, low and close prices Garman and Klass (1980)
                                                                                • \(\hat\sigma_t^2 = [\ln(O_t/C_{t-1})]^2 + \frac12[\ln(H_t/L_t)]^2 - (2\ln 2 - 1)[\ln(C_t/O_t)]^2\): the overnight return squared plus the Garman–Klass term
                                                                                • \(O_t\), \(H_t\), \(L_t\), \(C_t\): the open, high, low and close of day \(t\)
                                                                                • volatility is persistent and predictable (Chapter 5): the right ground for ML
                                                                              • HAR (heterogeneous autoregressive) model Corsi (2009): OLS on the log variance of the last day, week and month
                                                                                • \(\ln \bar\sigma^2_{t+1:t+5} = c + \beta_d \ln\hat\sigma_t^2 + \beta_w \ln\bar\sigma^2_{t-4:t} + \beta_m \ln\bar\sigma^2_{t-21:t} + u_t\); \(\bar\sigma^2_{a:b}\): the mean of \(\hat\sigma^2\) over days \(a\) to \(b\); \(u_t\): the error
                                                                                • ML models add 8 features: lags of the daily variance, the quarterly mean, the return, its negative part and its absolute value

                                                                              Realised volatility: HAR against ML (2/2)

                                                                              Slide 69 in the PDF

                                                                              The models are evaluated walk-forward from 2013, refitted every year, with a gap of 5 days between the training and the test data, because the 5-day targets overlap. Accuracy is measured relative to HAR by the out-of-sample \(R^2\):

                                                                              \[R^2_{\text{HAR}} = 1 - \frac{\sum_t (y_t - \hat y_t)^2}{\sum_t (y_t - \hat y_t^{\text{HAR}})^2} ,\]

                                                                              where \(y_t\) is the realised log variance and \(\hat y_t\), \(\hat y_t^{\text{HAR}}\) are the forecasts of the model and of HAR. A positive value means that the model has a smaller squared error than HAR.

                                                                              The second measure is the QLIKE loss of Patton (2011), the quasi-likelihood loss for variance forecasts used in Chapter 5:

                                                                              \[L(\sigma^2, h) = \frac{\sigma^2}{h} + \ln h ,\]
                                                                              (9.10)

                                                                              where \(\sigma^2\) is the realised variance and \(h\) the variance forecast. For a given \(\sigma^2\) the loss is smallest at \(h = \sigma^2\), and it penalises forecasts that are too low more than forecasts that are too high. Patton (2011) showed that it is robust to the noise in the variance proxy: it ranks forecasts in the same order as the true, unobserved variance would. Lower values are better, and the Diebold–Mariano test of Chapter 4 is applied to the daily differences in QLIKE.

                                                                              In brief

                                                                              • Yearly walk-forward from 2013 with a 5-day gap between training and test
                                                                              • Out-of-sample \(R^2\) against HAR: \(R^2_{\text{HAR}} = 1 - \sum_t (y_t - \hat y_t)^2 / \sum_t (y_t - \hat y_t^{\text{HAR}})^2\)
                                                                                • \(y_t\): the realised log variance; \(\hat y_t\), \(\hat y_t^{\text{HAR}}\): the forecasts of the model and of HAR; positive: better than HAR
                                                                              • QLIKE Patton (2011): \(L(\sigma^2, h) = \sigma^2/h + \ln h\), the quasi-likelihood loss for variances (Chapter 5); lower is better
                                                                                • \(\sigma^2\): the realised variance; \(h\): the variance forecast; for a given \(\sigma^2\) the loss is smallest at \(h = \sigma^2\) and penalises forecasts that are too low more
                                                                                • robust to the noise of the variance proxy: it ranks forecasts as the true variance would
                                                                              • Diebold–Mariano test (Chapter 4) on the daily QLIKE differences

                                                                              Results: S&P 500 and DAX, 2013–2026

                                                                              Slide 70 in the PDF

                                                                              The table reports, for the S&P 500 and the German DAX index, the out-of-sample \(R^2\) relative to HAR in percent, the mean QLIKE and the Diebold–Mariano statistic \(t\) against HAR, over 3,444 and 3,473 test days. A statistic below \(-1.96\) means that the model has a significantly lower QLIKE than HAR at the 5% level.

                                                                              The lasso and the MLP improve on HAR on both indices. On the S&P 500 the lasso has an \(R^2\) relative to HAR of 7.1% and a QLIKE of 0.400 against 0.414 for HAR (DM \(t = -5.12\)), and the MLP 8.3% and 0.398 (\(t = -3.96\)). On the DAX the gains in \(R^2\) are similar, 7.9% and 8.6%, but only the lasso lowers QLIKE significantly (\(t = -2.42\); for the MLP \(t = -1.69\)). The random forest and gradient boosting do not beat HAR: their \(R^2\) relative to HAR is between \(-0.5\)% and 3.5% and their QLIKE is not lower.

                                                                              The gain comes from the extra features, above all the negative part of the return, which captures the leverage effect of Chapter 5: falls in prices raise future volatility more than rises. The lasso and the MLP use these features through smooth functions, while trees cut a smooth relation into steps, which costs them the gain.

                                                                              In brief

                                                                              ModelS&P: \(R^2\) vs HAR (%)QLIKEDM \(t\)DAX: \(R^2\) vs HAR (%)QLIKEDM \(t\)
                                                                              HAR+0.00.414–+0.01.023–
                                                                              Lasso+7.10.400\(-\)5.12+7.91.015\(-\)2.42
                                                                              RF+1.00.4160.32+3.51.0240.03
                                                                              GB\(-\)0.50.4160.35+1.11.0280.75
                                                                              MLP+8.30.398\(-\)3.96+8.61.015\(-\)1.69
                                                                              • 3 444 and 3 473 test days; DM \(t < -1.96\): the model has a significantly lower QLIKE than HAR
                                                                              • The lasso and the MLP gain 7–9% of \(R^2\) over HAR and lower QLIKE significantly on the S&P 500; the random forest and GB do not beat HAR
                                                                              • The gain comes from the extra features (the negative return: the leverage effect of Chapter 5), used smoothly; trees cut a smooth relation into steps

                                                                              Volatility forecasts in 2020 and 2025

                                                                              Slide 71 in the PDF
                                                                                Question for the room Start the test period in 2020. Is the lasso still significantly more accurate than HAR on the S&P 500?
                                                                                Answer

                                                                                Yes: R² against HAR +7.4% and DM t = −3.42 on 1682 days; the MLP has +7.9% and t = −2.28.

                                                                                  Question for the room Show the lasso and the MLP for the S&P 500 in 2022 (left year). Does either beat HAR by QLIKE in that year?
                                                                                  Answer

                                                                                  Yes, both: QLIKE 1.536 (lasso) and 1.541 (MLP), against 1.553 for HAR. In 2025 HAR has 0.990, below both (0.997 and 1.002): one year does not decide the ranking.

                                                                                  The chart shows the annualised realised volatility of the S&P 500 over the next 5 days (black) and the forecasts of HAR, the random forest and gradient boosting in 2020 and 2025. In March 2020, during the COVID-19 crash, realised volatility reached about 100% per year; in early April 2025 it reached about 70%.

                                                                                  In both episodes all forecasts rise only after realised volatility has risen, and none of them reaches the peak. A forecast built from past variances reacts to a shock after it has happened, so the first days of a storm are always underestimated. After the peaks, the forecasts follow the decay of volatility closely. The three models are close to one another throughout, which agrees with the small differences in the table.

                                                                                  In brief

                                                                                  • S&P 500: annualised realised volatility of the next 5 days and the HAR, random forest and GB forecasts, 2020 and 2025
                                                                                  TSA_ch9_finance

                                                                                  Forecasting the sign of the next-day return: choice of the baseline

                                                                                  Slide 72 in the PDF

                                                                                  The second application is a classification problem. The target is \(1\{r_{t+1} > 0\}\), equal to 1 if tomorrow's return is positive and 0 otherwise. The features are the last five daily returns, the sums of returns over 5, 21 and 63 days and the volatility over 21 and 63 days. Three classifiers are compared, logistic regression (logit), a random forest and gradient boosting, evaluated walk-forward with yearly refits from 2008.

                                                                                  The right baseline is not 50%. Stock indices rise on more days than they fall, so the rule "always predict up", the majority class of the training data, is right on 54.4% of the S&P 500 days and 53.8% of the BET days in the test period. A classifier is useful only if it beats this rule. The accuracies are 53.8% for logit, 53.1% for the random forest and 52.5% for gradient boosting on the S&P 500, and 53.7%, 53.6% and 52.2% on the BET.

                                                                                  Accuracy depends on a threshold; the area under the ROC curve (AUC) does not. It is the probability that a randomly chosen up day receives a higher model score than a randomly chosen down day: 0.5 means no ability to rank days and 1 a perfect ranking. The models have AUC values between 0.505 and 0.533, barely above 0.5, so their ability to rank days by the probability of a rise is at most very weak.

                                                                                  In brief

                                                                                  • Classification: target \(1\{r_{t+1} > 0\}\) (1 if tomorrow's return is positive, 0 otherwise); features: the last five returns, sums over 5, 21, 63 days, volatility over 21 and 63 days
                                                                                    • models: logistic regression (logit), random forest, GB; yearly walk-forward from 2008
                                                                                  • The right baseline is not 50%: markets rise on more days than they fall
                                                                                    • “always up” (the majority class of the training data) is right on 54.4% of S&P 500 days and 53.8% of BET days
                                                                                  • Accuracy, S&P 500: logit 53.8%, random forest 53.1%, GB 52.5%; BET: 53.7%, 53.6%, 52.2%
                                                                                  • No model beats the baseline; AUC (area under the ROC curve) 0.505–0.533, close to 0.5: no ranking skill either
                                                                                    • AUC: the probability that a randomly chosen up day gets a higher model score than a randomly chosen down day; 0.5: no skill, 1: perfect ranking

                                                                                  Accuracy against the baseline

                                                                                  Slide 73 in the PDF
                                                                                    Question for the room Start the test period in 2016. What is the largest AUC?
                                                                                    Answer

                                                                                    0.541, the smallest 0.507; the up days are 54.6% (S&P 500) and 55.7% (BET). AUC stays close to 0.5.

                                                                                      Question for the room Raise the threshold to 0.55. What accuracy does GB reach on the S&P 500?
                                                                                      Answer

                                                                                      49.3%, below the 54.4% baseline: with the higher threshold the model forecasts “up” on only 40.4% of the days, while the market rises on 54.4% of them.

                                                                                      The chart compares the out-of-sample accuracy of the three classifiers with the baseline, from 2008 to 2026, on 4,705 S&P 500 days and 4,686 BET days. The golden bars are the baseline, the share of up days.

                                                                                      On both indices every classifier is below or at the baseline. On the BET, logit and the random forest are within 0.2 percentage points of it; on the S&P 500 all three are below it by 0.6 to 1.9 points. Gradient boosting, the most flexible model, has the lowest accuracy on both indices.

                                                                                      In brief

                                                                                      • Out-of-sample accuracy 2008–2026: 4 705 S&P 500 days and 4 686 BET days
                                                                                      TSA_ch9_finance

                                                                                      Interpreting the sign experiment

                                                                                      Slide 74 in the PDF

                                                                                      A model with "53% accuracy" sounds skilful, because it is right more often than a coin. Against the 54.4% of the rule "always up", it is worse than predicting the same answer every day. Gradient boosting on the S&P 500 is significantly worse than the baseline (p = 0.008): the flexible model fits patterns of the training years that are noise, and pays for them out of sample.

                                                                                      The result is the weak-form efficiency of Chapter 1 seen through classifiers: past returns contain almost no information about the sign of the next return. Together with the volatility results, it confirms what the statistical models of Chapters 1 and 5 showed: volatility is predictable, the direction is not.

                                                                                      In brief

                                                                                      • A model with “53% accuracy” sounds skilful; against the 54.4% of “always up” it is worse than doing nothing
                                                                                        • GB on the S&P 500 is significantly worse than the baseline (p = 0.008): flexible models fit noise
                                                                                      • Weak-form efficiency (Chapter 1): past returns contain almost no information about the sign of the next one
                                                                                      • Volatility is predictable, the direction is not: ML confirms what the statistical models of Chapters 1 and 5 showed

                                                                                      Recap: Volatility and the sign of returns

                                                                                      Slide 75 in the PDF

                                                                                      For realised volatility, the lasso and the MLP with extra features are more accurate than HAR, and the trees are not. For the sign of returns, the comparison is with the majority class, not with 50%, and no model beats it.

                                                                                      In brief

                                                                                      • Realised volatility: lasso and MLP with extra features beat HAR; trees do not
                                                                                      • Sign of returns: compare with the majority class, not with 50%; no model beats it

                                                                                      Section summary

                                                                                      Machine learning adds value in finance where there is structure to find: the lasso and the MLP gain 7 to 9% of \(R^2\) over HAR in volatility forecasting, thanks to the asymmetric response of volatility to returns. Where there is no structure, as in the direction of daily returns, flexible models only fit noise, and the majority-class baseline exposes it.

                                                                                      Landmark case studies: M4, M5 and Gu, Kelly and Xiu

                                                                                      The applications of this chapter use one series or a few. The three case studies below show the evidence at a much larger scale: two open forecasting competitions on tens of thousands of series, and a study of machine learning for the returns of US stocks.

                                                                                      The M competitions

                                                                                      Slide 76 in the PDF

                                                                                      Since 1982 Spyros Makridakis has organised open forecasting competitions, the M competitions. Every participating method forecasts the same series, and the organisers evaluate the forecasts on data that were hidden from the participants. This is the honest version of the walk-forward test: no participant can tune a method on the test data, and the results compare methods on equal terms.

                                                                                      Before M4, Makridakis, Spiliotis and Assimakopoulos (2018a) compared popular ML methods with eight statistical methods on 1,045 monthly series of the M3 competition. The ML methods were less accurate at every horizon and needed much more computation. The M4 competition of 2018 (Makridakis, Spiliotis and Assimakopoulos, 2020) then used 100,000 series of six frequencies, from yearly to hourly, and ranked 61 methods by the overall weighted average (OWA):

                                                                                      \[\text{OWA} = \frac12\Bigl(\frac{\text{sMAPE}}{\text{sMAPE}_{N2}} + \frac{\text{MASE}}{\text{MASE}_{N2}}\Bigr) .\]

                                                                                      Here \(N2\) is Naive2, the naive forecast of the seasonally adjusted series, and an OWA below 1 means more accurate than Naive2. The symmetric mean absolute percentage error is \(\text{sMAPE} = \frac{200}{H}\sum_{h=1}^{H} |y_{T+h} - \hat y_{T+h}| / (|y_{T+h}| + |\hat y_{T+h}|)\), in percent. The M5 competition of 2020 (Makridakis, Spiliotis and Assimakopoulos, 2022) used 42,840 hierarchical daily series of Walmart unit sales, with a horizon of 28 days.

                                                                                      In brief

                                                                                      • Since 1982 Spyros Makridakis has organised open forecasting competitions: all methods forecast the same series, evaluated by the organisers on hidden data
                                                                                        • the honest version of the walk-forward test: no participant can tune on the test data
                                                                                      • Makridakis, Spiliotis and Assimakopoulos (2018a): on 1 045 monthly series of M3, popular ML methods were less accurate than eight statistical methods at every horizon, and much slower
                                                                                      • M4 (2018), Makridakis, Spiliotis and Assimakopoulos (2020): 100 000 series of six frequencies, 61 methods, ranked by OWA (overall weighted average)
                                                                                        • \(\text{OWA} = \frac12\bigl(\text{sMAPE}/\text{sMAPE}_{N2} + \text{MASE}/\text{MASE}_{N2}\bigr)\); \(N2\): Naive2, the naive forecast of the seasonally adjusted series; below 1: better than Naive2
                                                                                        • sMAPE (symmetric mean absolute percentage error) \(= \frac{200}{H}\sum_{h=1}^{H} |y_{T+h} - \hat y_{T+h}| / (|y_{T+h}| + |\hat y_{T+h}|)\), in %
                                                                                      • M5 (2020), Makridakis, Spiliotis and Assimakopoulos (2022): 42 840 hierarchical daily series of Walmart unit sales, 28-day horizon

                                                                                      M4: the accuracy of all methods

                                                                                      Slide 77 in the PDF
                                                                                        Question for the room Set the weight of sMAPE to 1 (OWA from sMAPE only). Where does Comb rank?
                                                                                        Answer

                                                                                        23rd (OWA 0.926), against 19th; the winner is still method 118, a hybrid, with 0.839, and the best pure ML method falls to rank 27.

                                                                                        The chart plots the OWA of the 59 ranked methods of M4, submissions and benchmarks, against their rank, from the official evaluation file of the competition. The colours give the type of method: statistical, combinations of statistical methods, combinations of statistical and ML methods, combinations of ML methods, pure ML, hybrid and other. The dotted line is Naive2 (OWA = 1) and the dashed line the Comb benchmark; two methods with an OWA above 1.7 are outside the chart.

                                                                                        The ranking is dominated by combinations in the first places, with the single hybrid method in first place. The pure ML methods (red) are scattered: one ranks 23rd, the others rank 46th or lower, at the right end of the chart.

                                                                                        In brief

                                                                                        • Official evaluation file of M4 (github.com/Mcompetitions/M4-methods): OWA of the 59 ranked methods, submissions and benchmarks, coloured by type; 2 methods with OWA above 1.7 are off the chart
                                                                                        TSA_ch9_case_studies

                                                                                        Interpreting M4

                                                                                        Slide 78 in the PDF

                                                                                        The winner was the hybrid ES-RNN of Smyl (2020): exponential smoothing equations whose parameters are learned by an LSTM trained on all series together, which makes it a global model. It reached an OWA of 0.821 and a sMAPE of 11.37%. The Comb benchmark, the simple average of simple exponential smoothing (SES), Holt's linear trend and the damped trend method, reached an OWA of 0.898 and a sMAPE of 12.55%; Naive2 had a sMAPE of 13.56%.

                                                                                        The six pure ML methods did poorly: the best of them ranked 23rd, with an OWA of 0.915; none was more accurate than Comb and only one was more accurate than Naive2, as Makridakis, Spiliotis and Assimakopoulos (2018b) report. Twelve of the 17 most accurate methods were combinations. On short and heterogeneous series, the lesson is to combine statistical models or to let ML learn across series; pure ML applied series by series loses to simple benchmarks.

                                                                                        In brief

                                                                                        • Winner: the hybrid ES-RNN of Smyl (2020), exponential smoothing inside an LSTM trained on all series (a global model): OWA 0.821, sMAPE 11.37%
                                                                                          • Comb, the average of SES, Holt and damped trend: OWA 0.898, sMAPE 12.55%; Naive2: sMAPE 13.56%
                                                                                        • The 6 pure ML methods: the best ranked 23 (OWA 0.915); none beat Comb, only one beat Naive2 Makridakis, Spiliotis and Assimakopoulos (2018b)
                                                                                          • 12 of the 17 best methods were combinations
                                                                                        • Lesson: on short, heterogeneous series, combine statistical models, or let ML learn across series; pure local ML loses

                                                                                        M5: accuracy of gradient boosting on retail data

                                                                                        Slide 79 in the PDF

                                                                                        M5 attracted 5,507 teams from 101 countries. The data were the daily unit sales of 3,049 products in 10 Walmart stores, organised in a hierarchy of products, departments, categories, stores and states. The error measure was the weighted root mean squared scaled error (WRMSSE). For each series, the RMSSE divides the RMSE of the forecasts by the in-sample RMSE of the one-step naive forecast; the WRMSSE averages the RMSSE over all series with weights proportional to their sales value.

                                                                                        The winning method reached a WRMSSE of 0.520, 22.4% more accurate than the best benchmark, bottom-up exponential smoothing, with 0.671. Most of the top methods used LightGBM (Ke et al., 2017). The winner averaged LightGBM models pooled by store, by category and by department, with both recursive and direct strategies.

                                                                                        M5 had exactly the conditions under which ML wins: many related series, global models trained across them, and rich calendar and price features such as holidays, promotions and price changes. These conditions differ sharply from those of the single-series applications of this chapter.

                                                                                        In brief

                                                                                        M5: accuracy of gradient boosting on retail data A Walmart store: the M5 data are the daily unit sales of 3 049 products in 10 stores Photo: Walmart Corporate; CC BY 2.0; Wikimedia Commons
                                                                                        • 5 507 teams from 101 countries; error measure WRMSSE (weighted root mean squared scaled error)
                                                                                          • RMSSE: the RMSE of the forecasts over the in-sample RMSE of the one-step naive forecast
                                                                                          • WRMSSE: the average RMSSE, weighted by sales value
                                                                                          • the winner (WRMSSE 0.520) was 22.4% more accurate than the best benchmark, bottom-up exponential smoothing (0.671)
                                                                                        • Most top methods used LightGBM Ke et al. (2017)
                                                                                          • the winner averaged LightGBM models pooled by store, category and department, recursive and direct
                                                                                          • global models, rich calendar and price features, many related series: the conditions where ML wins

                                                                                        Gu, Kelly and Xiu (2020): machine learning for stock returns

                                                                                        Slide 80 in the PDF

                                                                                        Gu, Kelly and Xiu (2020) applied machine learning to the monthly returns of about 30,000 US stocks from 1957 to 2016, with 920 predictors built from stock characteristics, their interactions with macroeconomic variables and industry dummies. They compared 12 methods, from OLS to neural networks with five hidden layers, in a walk-forward design: 18 years of training, 12 years of validation and 30 years of test, with yearly refits.

                                                                                        The out-of-sample monthly \(R^2\) was at most 0.40%, reached by a network with three hidden layers. OLS with all predictors had an \(R^2\) of \(-3.46\)%: with so many predictors, unpenalised least squares overfits badly. Small as it is, the predictability has economic value. A long-short portfolio, which buys the decile of stocks with the highest forecasts and sells the decile with the lowest, reaches an annual Sharpe ratio of 1.35 with the four-layer network, against 0.61 for OLS-3, an OLS regression on three classic predictors. The Sharpe ratio is the mean annual return divided by its annual standard deviation.

                                                                                        In brief

                                                                                        Gu, Kelly and Xiu (2020): machine learning for stock returns Charles M. Harper Center, University of Chicago Booth School of Business Photo: Michael Barera (2013); CC BY-SA 4.0; Wikimedia Commons
                                                                                        • Gu, Kelly and Xiu (2020): monthly returns of about 30 000 US stocks, 1957–2016, 920 predictors
                                                                                          • 12 methods, from OLS to neural networks with five layers
                                                                                          • walk-forward: 18 years of training, 12 of validation, 30 years of test, refitted yearly
                                                                                        • Out-of-sample monthly \(R^2\): at most 0.40% (three-layer network); OLS with all predictors: \(-3.46\)%
                                                                                        • Small \(R^2\), large value: a long-short portfolio sorted on the forecasts reaches an annual Sharpe ratio of 1.35 (four layers), against 0.61 for OLS-3
                                                                                          • long-short: buy the decile with the highest forecasts, sell the decile with the lowest
                                                                                          • Sharpe ratio: the mean annual return divided by its annual standard deviation

                                                                                        Gu, Kelly and Xiu (2020): out-of-sample \(R^2\) and Sharpe ratios

                                                                                        Slide 81 in the PDF
                                                                                          Question for the room How many times larger is the Sharpe ratio of NN4 than that of OLS-3+H?
                                                                                          Answer

                                                                                          1.35/0.61 = 2.21 times, with a monthly R² of only 0.39%: a small predictability, used on thousands of stocks and months, supports a valuable portfolio.

                                                                                          The chart reproduces two published tables of the paper. The left panel gives the monthly out-of-sample \(R^2\) for all stocks (their Table 1), the right panel the annualised Sharpe ratio of the value-weighted long-short decile portfolio (their Table 7), for each method; the colours separate linear models, trees and neural networks. The suffix +H marks models estimated with the Huber loss, which reduces the influence of extreme returns.

                                                                                          The methods are OLS-3, an OLS regression on size, book-to-market and momentum; PLS and PCR, partial least squares and principal component regression; ENet, the elastic net, which combines the ridge and lasso penalties; GLM, a generalised linear model with splines; RF, the random forest; GBRT, gradient-boosted regression trees; and NN1 to NN5, networks with one to five hidden layers. Both panels show the same order: the nonlinear models, and the networks above all, have higher \(R^2\) and higher Sharpe ratios than the linear ones. An \(R^2\) below half a percent per month is small for a single forecast, but applied to thousands of stocks every month it is enough for a profitable portfolio.

                                                                                          In brief

                                                                                          • Published numbers: Table 1 (monthly out-of-sample \(R^2\), all stocks) and Table 7 (annualised Sharpe ratio of the value-weighted long-short decile portfolio); +H: Huber loss
                                                                                          • OLS-3: OLS on size, book-to-market and momentum; PLS, PCR: partial least squares, principal component regression; ENet: elastic net
                                                                                          • GLM: generalised linear model with splines; RF: random forest; GBRT: gradient-boosted regression trees; NN1–NN5: networks with 1–5 hidden layers
                                                                                          TSA_ch9_case_studies

                                                                                          Recap: Case studies

                                                                                          Slide 82 in the PDF

                                                                                          In M4, combinations and a hybrid global model were the most accurate, while pure ML applied series by series lost to simple benchmarks. In M5, global LightGBM models with calendar and price features won clearly. Gu, Kelly and Xiu (2020) found a small but real predictability in the cross-section of stock returns, where nonlinear models add economic value.

                                                                                          In brief

                                                                                          • M4: combinations and a hybrid global model win; pure local ML loses to simple benchmarks
                                                                                          • M5: global LightGBM models with calendar and price features win clearly
                                                                                          • Gu, Kelly and Xiu: tiny but real predictability in the cross-section of stocks; nonlinear models add value

                                                                                          Section summary

                                                                                          The large-scale evidence agrees with the applications of the chapter. Machine learning wins where many related series, global models and rich features are available, as in M5 and in the cross-section of stocks; on short, heterogeneous series analysed one at a time, combinations of statistical methods remain hard to beat.

                                                                                          Possible contribution of AI

                                                                                          The section points to foundation models, the extreme case of global models, and lists what an AI assistant can contribute to an ML forecasting project and what the analyst has to check, because the typical errors of such projects, leakage above all, produce no error message.

                                                                                          Foundation models: a pointer to Chapter 11

                                                                                          Slide 83 in the PDF

                                                                                          A foundation model for time series is a large neural network, usually a transformer, pre-trained on millions of series and then applied to a new series without retraining, which is called zero-shot forecasting. Chronos, TimesFM, Moirai and Lag-Llama are examples. They are global models taken to the extreme: one model for nearly every series. Chapter 11, a self-study chapter, presents them.

                                                                                          The rules of this chapter apply to them without change. The test data must be data the model has not seen in pre-training, which is hard to guarantee for public series; the evaluation is walk-forward; and the comparison uses the same benchmarks, the MASE and the Diebold–Mariano test.

                                                                                          In brief

                                                                                          • A foundation model for time series is a large network (a transformer) pre-trained on millions of series and used without retraining (zero-shot)
                                                                                            • the extreme global model: Chronos, TimesFM, Moirai, Lag-Llama (self-study, Chapter 11)
                                                                                          • The same rules apply: walk-forward on data the model has not seen in pre-training, the same benchmarks, MASE and DM

                                                                                          Possible contribution of AI

                                                                                          Slide 84 in the PDF

                                                                                          An assistant can help in three ways. It can write a first draft of the code: the feature table, the walk-forward loop and the evaluation table. It can give a second explanation of a method, such as boosting, the LSTM gates or conformal prediction. And it can explore many feature sets and hyperparameters at once, provided the exploration uses validation data only and the test data stay untouched.

                                                                                          The example prompt specifies every choice that matters for a leakage-free result: the features (lags, rolling means, calendar), the model (HistGradientBoostingRegressor), the strategy (direct, horizons 1 to 14), the evaluation (walk-forward against the seasonal naive forecast, with the MASE and the Diebold–Mariano test) and the rule that every preprocessing step is refitted inside each training window. A precise prompt produces code whose results can be checked against those of this chapter.

                                                                                          In brief

                                                                                          • Code: a first draft of the feature table, the walk-forward loop and the evaluation table
                                                                                          • Explanation: a second explanation of boosting, of the LSTM gates or of conformal prediction
                                                                                          • Exploration: many feature sets and hyperparameters at once (on validation data only)
                                                                                          • Example prompt
                                                                                            • Write Python code that builds lag, rolling-mean and calendar features for daily electricity load, trains HistGradientBoostingRegressor with the direct strategy for horizons 1-14, evaluates it walk-forward against the seasonal naive forecast with MASE and the Diebold-Mariano test, and refits every preprocessing step inside each training window.

                                                                                          Checks you must run

                                                                                          Slide 85 in the PDF

                                                                                          The checks follow the errors described in this chapter. Every feature must use data up to the origin only; the simplest check is to print a few rows of the table and verify the shifts of the lags and rolling windows by hand. The code must contain no random shuffling of the rows in cross-validation, such as KFold(shuffle=True), and no scaler or feature selection fitted on the full sample.

                                                                                          The benchmarks must be present: the naive or seasonal naive forecast and ETS or ARIMA, evaluated on the same origins. A comparison of forecasts needs a Diebold–Mariano test, and a classifier needs the majority-class baseline. The random seeds must be fixed, and the conclusion must survive a change of seed. Finally, every reference that an assistant cites must exist, which is checked through its DOI.

                                                                                          In brief

                                                                                          • Every feature uses data up to the origin only: check the shifts of lags and rolling windows on a printed table
                                                                                          • No KFold(shuffle=True), no scaler or feature selection fitted on the full sample
                                                                                          • The benchmarks are there: naive or seasonal naive, ETS or ARIMA, on the same origins
                                                                                          • The comparison has a DM test; the classification has the majority-class baseline
                                                                                          • Seeds are fixed and the result survives another seed
                                                                                          • Every cited reference exists: check the DOI

                                                                                          Section summary

                                                                                          An assistant can draft the feature table, the validation loop and the evaluation, and explain the methods, but its output is accepted only after the checks for leakage, benchmarks, significance tests, seeds and references.

                                                                                          Summary

                                                                                          The summary collects the main ideas of the chapter, the formulas to remember and three self-assessment questions.

                                                                                          Key takeaways

                                                                                          Slide 86 in the PDF

                                                                                          Forecasting with machine learning is supervised learning on a table of lags, rolling statistics and calendar features known at the origin. Only walk-forward validation is honest for time series; leakage produces spectacular and false scores. Ridge and lasso, random forests, gradient boosting, the MLP and the LSTM each have a complexity parameter that validation sets, and trees cannot extrapolate, so they model changes, not levels.

                                                                                          On the data of this chapter, linear models with good features match or beat ML on the electricity load and on inflation; ML helps for volatility, where the lasso and the MLP beat HAR; and no model predicts the sign of daily returns. ML wins with many related series, as in global models, the M4 hybrid and the LightGBM models of M5. Whatever the method, it is compared with benchmarks, with the MASE and the Diebold–Mariano test.

                                                                                          In brief

                                                                                          • ML forecasting = supervised learning on a table of lags, rolling statistics and calendar features known at the origin
                                                                                          • Walk-forward validation only; leakage produces spectacular and false scores
                                                                                          • Ridge and lasso, random forest, GB, MLP, LSTM: each has a complexity knob set by validation
                                                                                          • Trees cannot extrapolate: model changes, not levels
                                                                                          • Our data: linear models with good features match or beat ML on load and inflation; ML helps for volatility; nothing predicts the sign of returns
                                                                                          • ML wins with many related series (global models, M4 hybrid, M5 LightGBM); always compare with benchmarks, MASE and DM

                                                                                          The table collects the formulas of the chapter in the order in which they were introduced: the direct forecast from lags and calendar features of the target day; the ridge (9.2) and lasso penalties; the bias–variance decomposition (9.1); the boosting update (9.4); the MLP (9.5); the LSTM cell and output (9.6); the pinball loss (9.7); the split conformal interval (9.8); and the MASE, the MAE scaled by the in-sample MAE of the seasonal naive forecast with period \(m\).

                                                                                          In brief

                                                                                          QuantityFormula
                                                                                          Direct forecast\(\hat y_{T+h} = \hat f_h(y_T, \dots, y_{T-p+1}, \mathbf{z}_{T+h})\), \(\mathbf{z}_{T+h}\): calendar features
                                                                                          Ridge, lasso\(\min \sum_t (y_t - \mathbf{x}_t'\beta)^2 + \lambda\sum_j\beta_j^2\);   \(+ \lambda\sum_j|\beta_j|\)
                                                                                          Bias–variance\(E[(y_0 - \hat f(x_0))^2] = \text{bias}^2 + \Var(\hat f(x_0)) + \sigma^2\)
                                                                                          Boosting\(F_m(\mathbf{x}) = F_{m-1}(\mathbf{x}) + \nu\,g_m(\mathbf{x})\)
                                                                                          MLP\(\hat y = c + \sum_j v_j\,g(b_j + \mathbf{w}_j'\mathbf{x})\)
                                                                                          LSTM\(c_t = f_t \odot c_{t-1} + i_t \odot \tilde c_t\),   \(h_t = o_t \odot \tanh(c_t)\)
                                                                                          Pinball loss\(L_\tau(y, q) = \max(\tau(y - q), (\tau - 1)(y - q))\)
                                                                                          Split conformal\(\hat y \pm \hat q\),   \(\hat q = |e|_{(\lceil (n+1)(1-\alpha) \rceil)}\)
                                                                                          MASE\(\text{MAE} / \frac{1}{n-m}\sum_{t>m}|y_t - y_{t-m}|\)

                                                                                          Self-assessment

                                                                                          Slide 88 in the PDF

                                                                                          The first question concerns leakage. A 7-day moving average centred on \(t\) averages \(y_{t-3}, \dots, y_{t+3}\), so as a feature for \(y_{t+1}\) it contains the target itself and the two following values; it is not allowed. The trailing mean of \(y_{t-6}, \dots, y_t\) is the correct feature.

                                                                                          The second question concerns extrapolation. A random forest trained on GDP levels up to 2019 forecasts each level as a mean of training levels, so its forecast for 2024 cannot exceed the largest level seen before 2020, while GDP has grown since. The forest should be trained on growth rates. The third question concerns the baseline: a sign classifier with 53% accuracy is useful only if it beats, significantly, the share of up days in the test period, about 54% for the S&P 500. The next chapter, Chapter 10, presents state space models, the Kalman filter and Markov switching.

                                                                                          In brief

                                                                                          • Question: a 7-day moving average centred on \(t\) is used as a feature for \(y_{t+1}\). Is that allowed?
                                                                                            • Answer: no: it contains \(y_{t+1}, y_{t+2}, y_{t+3}\); use the trailing mean of \(y_{t-6}, \dots, y_t\)
                                                                                          • Question: a random forest trained on GDP levels up to 2019 forecasts 2024. What goes wrong?
                                                                                            • Answer: its forecast cannot exceed the largest level seen in training; model growth rates
                                                                                          • Question: a classifier predicts the sign of daily returns with 53% accuracy. Is it useful?
                                                                                            • Answer: only if it beats the share of up days in the test period (about 54% for the S&P 500), significantly
                                                                                          • Next: Chapter 10, state space models, the Kalman filter and Markov switching

                                                                                          Section summary

                                                                                          The chapter has placed machine learning inside the forecasting framework of the course: the same table of past information, the same walk-forward evaluation and the same benchmarks. Under these rules, flexible methods do not automatically win. They win when the data offer what they need, many related series, rich features and stable structure, and a well-specified linear model remains the benchmark to beat.

                                                                                          Daily mean electricity load of Romania (ENTSO-E, the extract of Chapter 4, 1 January 2022 – 30 June 2026). Daily S&P 500, DAX and BET prices (data/market, up to 18 September 2026). The Romanian HICP index (Eurostat prc_hicp_minr), read with the chapter’s loader on 4 November 2026 and cut at August 2026, the last month of the slides; it reproduces the numbers of the lecture exactly. The official M4 evaluation file (ch9_m4_owa.csv) and the published numbers of Gu, Kelly and Xiu (2020). The walk-forward forecasts of the models (ch9_load_ml.csv, ch9_inflation.csv, ch4_load_cv.csv) and the estimates that need the models to be retrained (random forest, gradient boosting, MLP, LSTM, lasso) are those of the chapter code, with the same seed; the browser recomputes the lag tables, the correlations, the regression trees, MASE, the Diebold–Mariano tests, the coverage of the intervals, RMSE, QLIKE, accuracy and AUC. Variants that do not appear in the lecture (another horizon, another noise level, another sample end, another interval level, another threshold) are extra runs of the chapter code with the same data and seed; the AR(1) models of the strategies block and the approximation of tanh by ReLU neurons are extra computations made in the browser.

                                                                                          References