Blockchain Applications

Lecture 8: Machine Learning, AI and Their Limits; Exam Review

Lecture 8: Machine Learning, AI and Their Limits; Exam Review

Daniel Traian PELE

Bucharest University of Economic Studies
IDA Institute Digital Assets
Blockchain Research Center
Master Blockchain for Business (MBB), FABIZ
26 November 2026

Lecture Question and Outline

Slide 2 in the PDF

The question of the lecture is whether machine learning and pretrained AI models forecast crypto returns and crypto risk better than the statistical models of Lectures 2–7, and how to test this honestly. Lecture 7 introduced the basic safeguards: separate training, validation and test periods, and a benchmark without the new information. This lecture adds three more: refitting the model step by step as time passes (walk-forward evaluation), accounting for searches over many rules, and checking whether a pretrained model has seen the test data.

The route has five parts. The first asks whether three classifiers, a logistic regression, a random forest and gradient boosting, can predict the direction of Bitcoin's daily return. The second shows what happens when 810 trading rules are tried and the best one is kept. The third compares classical, ML and pretrained models for Value-at-Risk. The fourth lists the limits of ML and AI in crypto finance, and the fifth reviews the course for the exam. All estimates use daily Bitcoin prices and volumes from EODHD, 2015 – 23 October 2026.

In brief

  • Question: machine learning (ML) and pretrained artificial intelligence (AI) models are now standard tools. Do they forecast crypto returns and crypto risk better than the statistical models of Lectures 2–7, and how do we find out without fooling ourselves?
  • Outline
    • ML forecasts of the direction of Bitcoin: logit, random forest, gradient boosting
    • data snooping: what happens when we try 810 trading rules
    • risk forecasts: classical, ML and pretrained (foundation) models for Value-at-Risk
    • the limits of ML and AI in crypto finance
    • exam review: what you should be able to interpret
  • From Lecture 7: train, validation and test periods, and a benchmark without the new information; today adds walk-forward refitting, many-rule searches and pretrained models
  • Data: EOD Historical Data (EODHD) daily Bitcoin prices and volumes, 2015 – 23 October 2026

Learning Outcomes

Slide 3 in the PDF
  • Set up a walk-forward evaluation and explain why in-sample accuracy is not evidence
  • Read accuracy, the Brier score, the area under the curve and the Pesaran–Timmermann test against the right benchmark
  • Explain data snooping and the deflated Sharpe ratio
  • Compare Value-at-Risk forecasts by breach rate and by quantile score, and explain why these can disagree
  • Name the specific risks of pretrained models: look-ahead through pretraining, survivorship, reproducibility
  • Interpret the main results of Lectures 1–8 in the exam format

Abbreviations

Slide 4 in the PDF
  • AI artificial intelligence
  • AUC area under the ROC curve
  • BTC Bitcoin
  • CRIX CRypto IndeX; VCRIX its volatility index
  • DM Diebold–Mariano (test)
  • DSR deflated Sharpe ratio
  • EODHD EOD Historical Data
  • ES expected shortfall
  • ETF exchange-traded fund
  • EWMA-N exponentially weighted moving average, Normal quantile
  • GARCH-t generalised autoregressive conditional heteroskedasticity, Student-\(t\) errors
  • HAC heteroskedasticity- and autocorrelation-consistent
  • HAR heterogeneous autoregressive
  • HS historical simulation
  • LLM large language model
  • LR likelihood ratio (test)
  • ML machine learning
  • ROC receiver operating characteristic
  • MOM(\(k\)) \(k\)-day momentum rule
  • MA(\(s\),\(l\)) moving-average crossover, \(s\) and \(l\) days
  • PSR probabilistic Sharpe ratio
  • PT Pesaran–Timmermann test
  • QGBM gradient-boosted quantile trees
  • QReg linear quantile regression
  • QS quantile score (pinball loss)
  • SHAP Shapley additive explanations
  • SR Sharpe ratio
  • VaR Value-at-Risk
  • XGBoost extreme gradient boosting

Reading and Tools

Slide 5 in the PDF

The tools of the lecture come from a small set of papers. Koenker & Bassett (1978) introduced quantile regression, the basis of the quantile models for Value-at-Risk, and Pesaran & Timmermann (1992) the test of directional accuracy. White (2000) proposed the "reality check" for data snooping, and Bailey & López de Prado (2014) the deflated Sharpe ratio. Gneiting & Resin (2023) study calibration diagnostics and the optimal shift of a forecast, the subject of the Appendix.

The pretrained forecasting model is Chronos-2 of Ansari et al. (2025); the explanations of model predictions use SHAP (Lundberg & Lee, 2017). The zones for backtesting a Value-at-Risk model come from the market-risk standard of the Basel Committee on Banking Supervision (Basel Committee, 2019). Two video units on Quantinar accompany the lecture. The methods are available in Python: `scikit-learn` fits the logit and the random forest, `xgboost` gradient boosting, `shap` the explanations, `arch` GARCH models, `statsmodels` quantile regression, and `chronos-forecasting` runs Chronos-2; the notebook of Lecture 8 applies them to the Bitcoin data of the lecture.

In brief

Even when no exploitable forecasting relation exists, looking long enough and hard enough at a given set of data will often reveal one or more forecasting models that look good, but are in fact useless.

— White (2000)

Machine learning (ML) and pretrained artificial intelligence (AI) models are now standard tools in finance. Software libraries fit a random forest or a gradient-boosting model in a few lines of code, and pretrained forecasting models return a full forecast distribution for any series without being fitted to it. In crypto markets, where data are public and plentiful, claims that such models predict prices are common.

The lecture asks two practical questions. Do these models forecast crypto returns and crypto risk better than the statistical models of Lectures 2 to 7? And how can an analyst find out without fooling himself or herself? The second question is the harder one. A flexible model can fit the past almost perfectly, a search over many trading rules always finds a winner, and a pretrained model may already have seen the period on which it is tested. Each of these makes a model look better than it is.

The data are daily Bitcoin prices and trading volumes from EOD Historical Data (EODHD), from 2015 to 23 October 2026. The lecture uses them for three tests: forecasting the direction of tomorrow's return, selecting a trading rule among 810 candidates, and forecasting Value-at-Risk with six models. It then lists the limits of ML and AI in crypto finance and closes with a review of the course for the exam.

Accuracy of three classifiers of the direction of Bitcoin's daily return, test period 1 January 2020 – 22 October 2026. Left: share of days classified correctly in the training data (in-sample, dark blue) and on new days (out-of-sample, orange); the dashed line is the 50.6% of a forecast that always predicts "up". Right: mean absolute contribution of each input to the predictions of the gradient-boosting model, measured by SHAP over the last 400 test days.
Accuracy of three classifiers of the direction of Bitcoin's daily return, test period 1 January 2020 – 22 October 2026. Left: share of days classified correctly in the training data (in-sample, dark blue) and on new days (out-of-sample, orange); the dashed line is the 50.6% of a forecast that always predicts "up". Right: mean absolute contribution of each input to the predictions of the gradient-boosting model, measured by SHAP over the last 400 test days.

What you will be able to do

  • Set up a walk-forward evaluation and explain why in-sample accuracy is not evidence.
  • Read accuracy, the Brier score, the area under the curve and the Pesaran–Timmermann test against the right benchmark.
  • Explain data snooping and the deflated Sharpe ratio.
  • Compare Value-at-Risk forecasts by breach rate and by quantile score, and explain why these can disagree.
  • Name the specific risks of pretrained models: look-ahead through pretraining, survivorship, reproducibility.
  • Interpret the main results of Lectures 1–8 in the exam format.

The Appendix holds optional material on correcting a risk forecast with its own past errors.

Lecture plan

  1. Machine-Learning Forecasts of Bitcoin's DirectionLogit, random forest and gradient boosting in a walk-forward test; accuracy, Brier score, AUC and the Pesaran–Timmermann test; trading the signals.
  2. Data SnoopingWhy the best of many rules looks good by chance; the probabilistic and deflated Sharpe ratio; 810 trading rules.
  3. Risk Forecasts: Classical, ML and PretrainedSix Value-at-Risk models; breach rate against quantile score; jumps of tree models; look-ahead in pretrained models.
  4. The Limits of ML and AI in Crypto FinanceSeven limits, and where ML does help.
  5. Exam ReviewThe exam format, the main tools of Lectures 1–8 and a model answer.
  6. SummaryThe main numbers of the lecture and five interpretation exercises.
  7. Appendix: Further MaterialCorrecting a forecast with its own errors: theory, simulation and the Bitcoin forecasts.

Machine-Learning Forecasts of Bitcoin's Direction

The simplest forecasting task for a trader is the direction of tomorrow's price: up or down. The section sets up this task for three classifiers of increasing flexibility, evaluates them on almost seven years of new data with several measures of skill, and asks whether trading on their signals would have beaten holding Bitcoin.

The Prediction Problem

Slide 6 in the PDF

The target is the direction of tomorrow's return, \(y_{t+1} = \mathbb{1}\{r_{t+1} > 0\}\), which equals 1 on an up day and 0 otherwise. The 13 inputs, or features, \(x_t\) are all known at today's close: the returns of the last three days, the 5-, 20- and 60-day momentum (the past return over that period), the 5- and 20-day volatility, changes in trading volume, the gaps between the price and its 50- and 200-day averages, and the day of the week.

Three classifiers of increasing flexibility map the features into a probability of an up day. The logistic regression (logit) is linear in the features,

\[P(y_{t+1} = 1 \mid x_t) = \Lambda(\beta_0 + x_t'\beta), \qquad \Lambda(u) = \frac{1}{1 + e^{-u}}\]
(8.1)

and is fitted with a ridge penalty, which shrinks the coefficients towards zero to limit overfitting. A random forest averages \(B = 500\) decision trees, each grown on a bootstrap sample of the days and using random subsets of the features. Gradient boosting, here in the XGBoost implementation (extreme gradient boosting), adds trees one at a time, each fitted to the errors of the current model:

\[F_m(x) = F_{m-1}(x) + \eta\, h_m(x)\]
(8.2)

where \(h_m\) is the new tree, fitted to the gradient of the log-loss, and \(\eta = 0.05\) the learning rate; the model has 300 trees of depth 3.

The evaluation is walk-forward. At a forecast origin \(\tau\), each model is trained on all data up to \(\tau\) and predicts the next 63 days; then \(\tau\) moves forward by 63 days and the model is refitted, 40 times in all. The first origin trains on data up to 31 December 2019 and predicts 1 January – 3 March 2020; the second trains on data up to 3 March 2020, now including those 63 realised days, and predicts 4 March – 5 May 2020. The test period runs from 1 January 2020 to 22 October 2026, \(T = 2{,}487\) days. Each forecast uses only information available before its target day; realised test days join the training data later, which is not leakage, because a real forecaster would have them too.

In brief

  • Target: tomorrow's direction \(y_{t+1} = \mathbb{1}\{r_{t+1} > 0\}\); features \(x_t\) known at today's close: returns of the last three days, 5-, 20- and 60-day momentum, 5- and 20-day volatility, volume changes, gaps to the 50- and 200-day averages, day of the week (13 features)
  • Three classifiers of increasing flexibility:
    • logistic regression (logit): \(\;P(y_{t+1} = 1 \mid x_t) = \Lambda(\beta_0 + x_t'\beta)\), \(\Lambda(u) = 1/(1 + e^{-u})\), with a ridge penalty
    • random forest: average of \(B = 500\) trees, each grown on a bootstrap sample with random feature subsets
    • gradient boosting (XGBoost): additive trees, \(F_m(x) = F_{m-1}(x) + \eta\, h_m(x)\), each \(h_m\) fitted to the gradient of the log-loss; \(\eta = 0.05\), 300 trees of depth 3
  • Walk-forward design: at origin \(\tau\), train on all data up to \(\tau\), predict the next 63 days; move \(\tau\) forward 63 days and refit (40 origins); test 1 January 2020 – 22 October 2026 (\(T = 2\,487\) days)
    • origin 1: train to 31 December 2019, predict 1 January – 3 March 2020
    • origin 2: train to 3 March 2020 (now including those 63 realised days), predict 4 March – 5 May 2020
  • Each forecast uses only information available before its target day; realised test days later join the training data, which is not leakage

Measuring Predictive Skill

Slide 7 in the PDF

The obvious measure is accuracy, the share of days on which the predicted direction is right, \(T^{-1}\sum_t \mathbb{1}\{\hat y_t = y_t\}\). It must be compared with the base rate: Bitcoin rose on 50.6% of the test days, so the forecast "always up" already scores 50.6%. A model is useful only if it beats that number.

Accuracy uses only the predicted direction. The Brier score uses the predicted probabilities \(p_t\):

\[\mathrm{BS} = \frac1T\sum_{t=1}^{T} (p_t - y_t)^2\]
(8.3)

Lower is better. Two constant forecasts serve as benchmarks. The constant \(p_t = \bar y\), the share of up days in the test period, gives \(\bar y(1-\bar y) = 0.2500\), but it uses the test data and was not available at forecast time. The constant estimated from the past only, the share of up days in the training data at each origin (between 0.526 and 0.556), gives 0.2510.

The area under the receiver operating characteristic curve (AUC) measures whether the model ranks days correctly: it is the probability that a randomly chosen up day gets a higher \(p_t\) than a randomly chosen down day; 0.5 means no skill. The Pesaran–Timmermann (PT) test of Pesaran & Timmermann (1992) compares the hit rate \(\hat P\) with the hit rate \(P^* = \bar y\, \bar{\hat y} + (1-\bar y)(1-\bar{\hat y})\) expected if forecasts and outcomes were independent, where \(\bar{\hat y}\) is the share of "up" forecasts:

\[\mathrm{PT} = \frac{\hat P - P^*}{\sqrt{\widehat{V}(\hat P) - \widehat{V}(P^*)}} \;\overset{a}{\sim}\; N(0,1)\]
(8.4)

The forecasts are also traded. The strategy is long Bitcoin when \(p_t > 0.5\) and in cash otherwise, with a cost of 10 basis points (bp, hundredths of a percent) each time the position changes. Its annualised Sharpe ratio (SR) is \(\sqrt{365}\,\bar g / s_g\), where \(\bar g\) and \(s_g\) are the mean and standard deviation of its daily returns. An estimated SR has sampling error; over \(Y\) years,

\[\mathrm{se}(\widehat{\mathrm{SR}}) \approx \sqrt{\frac{1 + \mathrm{SR}^2/2}{Y}}\]
(8.5)

which is about 0.4 for the \(Y = 6.8\) years of the test period. Two Sharpe ratios that differ by less than 0.4 are therefore hard to tell apart.

In brief

  • Accuracy \(= T^{-1}\sum_t \mathbb{1}\{\hat y_t = y_t\}\) must beat the base rate: Bitcoin rose on 50.6% of test days, so “always up” already scores 50.6%
  • Brier score of the probabilities \(p_t\): \(\;\mathrm{BS} = T^{-1}\sum_t (p_t - y_t)^2\); two constant benchmarks
    • ex post: \(p_t = \bar y\) of the test period gives \(\bar y(1-\bar y) = 0.2500\) (not available at forecast time)
    • from the past only: \(p_t\) = share of up days in the training data at each origin (0.526–0.556) gives 0.2510
  • Area under the receiver operating characteristic curve (AUC): probability that a random up-day gets a higher \(p_t\) than a random down-day; 0.5 = no skill
  • Pesaran–Timmermann (PT) test: compares the hit rate \(\hat P\) with the hit rate expected if forecasts and outcomes were independent, \(P^* = \bar y\, \bar{\hat y} + (1-\bar y)(1-\bar{\hat y})\):
    \[ \mathrm{PT} = \frac{\hat P - P^*}{\sqrt{\widehat{V}(\hat P) - \widehat{V}(P^*)}} \;\overset{a}{\sim}\; N(0,1) \]
  • Trading: long when \(p_t > 0.5\), otherwise cash; 10 basis points (bp) cost per change of position; annualised Sharpe ratio (SR) \(= \sqrt{365}\,\bar g / s_g\)
  • Sampling error of an annualised SR over \(Y\) years: \(\mathrm{se} \approx \sqrt{(1 + \mathrm{SR}^2/2)/Y}\), about 0.4 for \(Y = 6.8\)

Results, 2020–2026

Slide 8 in the PDF

The table compares the three classifiers with the "always up" forecast, which as a trading strategy is buy-and-hold. The first point is the gap between in-sample and out-of-sample accuracy. In the training data the logit classifies 55.8% of days correctly, the random forest 64.6% and XGBoost 78.3%. On new days the three reach only 51.7%, 51.6% and 51.3%, between 0.7 and 1.1 percentage points above the base rate of 50.6%. The more flexible the model, the larger the gap: XGBoost fits 78% of past days and 51% of new ones. This is overfitting: the model learns noise of the training data that does not repeat.

The second point is whether the small gain in accuracy is real. Only the logit passes the PT test at 5% (one-sided p-value 0.03); the random forest (p 0.07) and XGBoost (p 0.13) do not. The Brier scores are all worse than both constants: 0.2515, 0.2514 and 0.2611 against 0.2500 ex post and 0.2510 from the past. The raw probabilities therefore do not improve on a constant forecast, although the logit ranks days slightly better than chance (AUC 0.525).

The trading columns tell the same story. The Sharpe ratios of the strategies, 0.63–0.81, exceed the 0.59 of buy-and-hold, but the differences are far below one standard error of about 0.4. The logit is long 85% of the time, so it is mostly buy-and-hold. The maximum drawdown, the largest fall of the strategy's wealth from a previous peak, computed as \(1 - e^{-D_{\log}/100}\) with \(D_{\log}\) the fall in cumulative log return in percent (Lecture 2), is 68.6–80.0% for the strategies against 76.6% for buy-and-hold.

Finally, accuracy by year falls from 55% in 2020 to 46–51% in 2025–2026. Whatever the models learnt in the early years does not persist.

In brief

accuracy (%)AUCBrierPTshareSRmax. draw-
in-sampleout-of-sample(p)longdown (%)
Always up / buy and hold–50.60.5000.2500–100%0.5976.6
Logit55.851.70.5250.25150.0385%0.7268.6
Random forest64.651.60.5140.25140.0778%0.8180.0
XGBoost78.351.30.5080.26110.1365%0.6376.9
  • Max. drawdown: largest fall of the strategy's wealth from a previous peak, \(1 - e^{-D_{\log}/100}\) with \(D_{\log}\) the fall in cumulative log return (Lecture 2)
  • Out-of-sample accuracy is 0.7–1.1 percentage points above the base rate; only the logit passes the PT test at 5% (one-sided)
  • Every Brier score is worse than both constants (0.2500 ex post, 0.2510 from the past): the raw probabilities do not improve on a constant, although the logit ranks days slightly better than chance (AUC 0.525)
  • The more flexible the model, the larger the gap between in-sample and out-of-sample accuracy: XGBoost fits 78% of past days and 51% of new ones (overfitting)
  • Sharpe ratios of 0.63–0.81 against 0.59: differences far below one standard error (\(\approx 0.4\)); the logit is long 85% of the time, so it is mostly buy-and-hold
  • Accuracy by year falls from 55% (2020) to 46–51% (2025–2026): whatever was learnt does not persist
  • Source: own computation on EODHD daily Bitcoin prices and volumes, 2015 – 23 October 2026; test days 2020–2026

Worked Example: the Pesaran–Timmermann Test for the Logit

Slide 9 in the PDF

The PT test is easier to read once it has been computed by hand. The inputs come from the 2,487 test days: Bitcoin rose on a share \(\bar y = 0.506\) of them, the logit forecast "up" on a share \(\bar{\hat y} = 0.854\), and it was right on 1,287 days, a hit rate of \(\hat P = 1{,}287/2{,}487 = 0.5175\). Of these hits, 1,092 were correct "up" forecasts and only 195 correct "down" forecasts.

The first step computes the hit rate that forecasts unrelated to the outcome would achieve with the same shares: \(P^* = \bar y\,\bar{\hat y} + (1-\bar y)(1-\bar{\hat y}) = 0.506 \times 0.854 + 0.494 \times 0.146 = 0.5044\). This, not 50%, is the right benchmark: a forecaster who says "up" 85% of the time in a market that rises on 51% of days is right about half the time by construction.

The second step computes two variances: \(\widehat V(\hat P) = P^*(1 - P^*)/T = 1.005 \times 10^{-4}\), and \(\widehat V(P^*) = 0.505 \times 10^{-4}\), from the formula of Pesaran & Timmermann (1992). Their difference gives the standard error \(\sqrt{(1.005 - 0.505) \times 10^{-4}} = 0.00707\). The third step divides the excess hit rate by this standard error: \(\mathrm{PT} = (0.5175 - 0.5044)/0.00707 = 1.85\), with a one-sided p-value of 0.032, significant at 5%.

The size of the effect is small. The excess hit rate of 1.3 percentage points means about 33 more correct days out of 2,487 than independent forecasts would give. Most of the logit's hits are up days, which it gets by being long most of the time.

In brief

  • Inputs, \(T = 2\,487\) test days: share of up days \(\bar y = 0.506\); share of “up” forecasts \(\bar{\hat y} = 0.854\); hit rate \(\hat P = 1\,287/2\,487 = 0.5175\) (1 092 correct “up” and 195 correct “down”)
  • Step 1, hit rate if forecasts ignored the outcome: \(P^* = \bar y\,\bar{\hat y} + (1-\bar y)(1-\bar{\hat y}) = 0.506 \times 0.854 + 0.494 \times 0.146 = 0.5044\)
  • Step 2, variances:
    • \(\widehat V(\hat P) = P^*(1-P^*)/T = 1.005 \times 10^{-4}\)
    • \(\widehat V(P^*) = 0.505 \times 10^{-4}\) (from the formula of Pesaran & Timmermann (1992))
    • standard error \(\sqrt{1.005 - 0.505} \times 10^{-2} = 0.00707\)
  • Step 3, statistic: \(\mathrm{PT} = (0.5175 - 0.5044)/0.00707 = 1.85\); one-sided p \(= 0.032\): significant at 5%
  • Reading: the benchmark is \(P^* = 50.4\%\), not 50%; a model that is long 85% of the time earns most of its hits from up days
  • The excess hit rate is \(1.3\) percentage points: about 33 more correct days out of 2 487 than independent forecasts would give
  • Source: own computation on EODHD daily Bitcoin prices and volumes, 2015 – 23 October 2026; test days 2020–2026

Accuracy and the Base Rate

Slide 10 in the PDF

The results table averages over almost seven years. The chart splits the out-of-sample accuracy by calendar year: the bars give the accuracy of the logit (dark blue), the random forest (green) and XGBoost (purple), and the red line the share of up days in each year, which is the accuracy of "always up" in that year.

Accuracy moves with the base rate. In 2020, when Bitcoin rose on 56.3% of days, the three models reached 55–57%; in 2022, when it rose on only 46.3% of days, they reached 48.5–50.4%. The reason is that the models are long most of the time, between 58% and 95% of the days in every year, so they are right when the market rises and wrong when it falls.

Against the base rate of each year, each model is ahead in only four or five of the seven years. A forecaster with a stable edge would beat the base rate year after year; these models do not, and their accuracy in 2025 and 2026 is at or below 51%.

In brief

Accuracy and the Base Rate
  • Bars: out-of-sample accuracy by calendar year; red line: share of up days in that year, the accuracy of “always up”
  • Accuracy is high when the market rises (2020: 55–57%, base rate 56.3%) and low when it falls (2022: 48.5–50.4%, base rate 46.3%)
  • The models are long 58–95% of days in every year, so their accuracy moves with the base rate
  • Each model beats the base rate in only 4 or 5 of the 7 years: no year-by-year evidence of a stable edge
  • Source: own computation on EODHD daily Bitcoin prices and volumes, 2015 – 23 October 2026; test days 2020–2026
BA_L8_acc_year

Calibration of the Predicted Probabilities

Slide 11 in the PDF

The Brier score showed that the probabilities are worse than a constant. The reliability diagram on the left shows why. The test days are grouped by their forecast probability \(p_t\) into bins of width 0.03 around 0.5, open at both ends, and for each bin the share of up days is plotted against the mean forecast. A calibrated forecast lies on the dashed diagonal: among the days on which it says 0.55, 55% are up days. The right panel shows the histograms of the forecast probabilities.

The logit is cautious: 90% of its probabilities lie between 0.475 and 0.607. Its points lie near, but below, the diagonal at the top: when it says 0.59 on average, 53% of the days are up days. XGBoost is the opposite. It spreads its probabilities from 0.36 to 0.71 (5% and 95% quantiles), yet up days are 49% in its lowest bin and 51% in its highest. Its forecasts are confident and almost uninformative, which is exactly what the Brier score penalises: XGBoost has the worst Brier score of the three, 0.2611.

A model can therefore have an accuracy above 50% and still produce probabilities that should not be used as probabilities, for example to size positions. Calibration can be checked, and corrected, only on data that were not used to fit the model.

In brief

Calibration of the Predicted Probabilities
  • Left: test days grouped by forecast probability \(p_t\) (bins of width 0.03 around 0.5); share of up days in each bin against the mean \(p_t\). Calibrated forecasts lie on the dashed line
  • Logit: 90% of its \(p_t\) lie between 0.475 and 0.607; when it says 0.59 on average, up days are 53%
  • XGBoost spreads its \(p_t\) from 0.36 to 0.71 (5% and 95% quantiles), but up days are 49% in its lowest bin and 51% in its highest: confident and uninformative, hence the worst Brier score (0.2611)
  • Source: own computation on EODHD daily Bitcoin prices and volumes, 2015 – 23 October 2026; test days 2020–2026
BA_L8_calib

Overfitting and Feature Contributions

Slide 12 in the PDF
    Question for the room The ▶ button goes through the 40 walk-forward origins. At how many of them is the out-of-sample accuracy of XGBoost, on the next 63 days, above its in-sample accuracy on the training data?
    Answer

    At none of the 40. In-sample, XGBoost fits 74–85% of past days; out of sample its accuracy on a 63-day block ranges from 38% to 67%. The random forest also never does better out of sample; the logit, which fits only 54–58% in sample, does better at 13 of the 40 origins. A flexible model learns the noise of the training days, so its in-sample accuracy says little about new days. Out of sample, XGBoost beats "always up" in 15 of the 40 blocks, and over all 2,487 test days it scores 51.3% against a base rate of 50.6%.

    The left panel shows the gap between in-sample (dark blue) and out-of-sample (orange) accuracy for the three classifiers, with the dashed line at the base rate of 50.6%. The in-sample bars grow with the flexibility of the model; the out-of-sample bars stay just above the dashed line.

    The right panel explains the predictions of XGBoost with Shapley additive explanations (SHAP) of Lundberg & Lee (2017). SHAP splits each prediction into a baseline and contributions of the features,

    \[f(x) = \phi_0 + \sum_j \phi_j(x)\]
    (8.6)

    where \(\phi_j\) is the average marginal contribution of feature \(j\) over all orders in which the features can be added to the model. The bars show the mean absolute contribution of each feature over the last 400 test days. The largest contributions come from today's return and the 5-day volatility; the day of the week ranks fourth.

    SHAP explains the model, not the market. A feature with a large \(|\phi_j|\) in a model that has no forecasting skill is not evidence that the feature predicts returns: it only shows what the model relies on.

    In brief

    • Shapley additive explanations (SHAP): the prediction is split into feature contributions, \(f(x) = \phi_0 + \sum_j \phi_j(x)\), with \(\phi_j\) the average marginal contribution of feature \(j\) over all orderings of the features
    • The largest contributions come from today's return and 5-day volatility; the day of the week ranks fourth
    • SHAP explains the model, not the market: a large \(|\phi_j|\) for a model with no skill is not evidence that the feature predicts returns
    • Source: own computation on EODHD daily Bitcoin prices and volumes, 2015 – 23 October 2026; test days 2020–2026
    BA_L8_ml_acc

    Trading the Signals

    Slide 13 in the PDF
      Question for the room The ▶ button raises the cost per change of position from 0 to 50 basis points. Which strategy loses the most Sharpe ratio?
      Answer

      XGBoost: its Sharpe ratio falls from 0.89 (no cost) to 0.63 (10 bp) and −0.43 (50 bp); it changes position about 125 times a year. The random forest (81 changes a year) falls from 0.97 to 0.19 and the logit (70 a year) from 0.85 to 0.22. Buy-and-hold pays the cost once and stays at 0.59. XGBoost falls below buy-and-hold at about 12 bp, the logit at about 21 bp and the random forest at about 25 bp. The Sharpe ratio before costs does not rank the strategies after costs: turnover does.

      The chart plots the cumulative log return, after costs of 10 bp, of buy-and-hold and of the three strategies from January 2020 to October 2026. The gains over buy-and-hold come in a few episodes, in 2020–2021 and in parts of 2022, when the strategies were partly in cash during falls; between these episodes the curves move in parallel with buy-and-hold.

      A strategy that is long 65–85% of the time inherits the market's drawdowns: the maximum drawdowns are 69–80% against 77% for buy-and-hold. On the log scale of the chart these drawdowns are falls of 116–161 log points (145 for buy-and-hold). A fall of more than 100 log points is not a loss of more than 100% of wealth: a fall of 145 log points leaves \(e^{-1.45} = 23\%\) of the peak wealth, a drawdown of 77%.

      In brief

      • The gains over buy-and-hold come in a few episodes (2020–2021, parts of 2022); in between, the curves move in parallel with buy-and-hold
      • A strategy that is long 65–85% of the time inherits the market's drawdowns: maximum drawdowns of 69–80% against 77% for buy-and-hold
      • The chart plots cumulative log returns: these drawdowns are falls of 116–161 log points (145 for buy-and-hold), which are less than 100% of wealth
      • Source: own computation on EODHD daily Bitcoin prices and volumes, 2015 – 23 October 2026; test days 2020–2026
      BA_L8_ml_equity

      Section summary

      Machine-learning classifiers predict the direction of Bitcoin barely better than "always up": 51.3–51.7% out of sample against a base rate of 50.6%, with Brier scores worse than a constant. In-sample accuracy grows with flexibility and says nothing about forecasting skill, and the trading gains are within one standard error of buy-and-hold.

      Data Snooping

      Data snooping means using the same data both to search for a model and to evaluate it. If many rules are tried and the best is reported, its performance is overstated, even when none of the rules has any skill. The section quantifies this effect and applies the corrections to a search over 810 trading rules.

      Data Snooping: the Deflated Sharpe Ratio and the Reality Check

      Slide 14 in the PDF

      Suppose \(N\) trading rules are tested and none has skill: the true Sharpe ratio of each is zero. The estimated Sharpe ratios still vary by chance, and the best of them is positive and grows with \(N\). For \(N\) independent rules, the expected maximum is approximately

      \[\mathrm{SR}_0 = \sqrt{V[\widehat{\mathrm{SR}}_n]}\left[(1-\gamma)\,\Phi^{-1}\!\left(1 - \tfrac1N\right) + \gamma\, \Phi^{-1}\!\left(1 - \tfrac{1}{N e}\right)\right]\]
      (8.7)

      where \(V[\widehat{\mathrm{SR}}_n]\) is the variance of the estimated Sharpe ratios across rules, \(\Phi^{-1}\) the quantile function of the standard Normal distribution and \(\gamma \approx 0.5772\) the Euler–Mascheroni constant. Here SR is in daily units, the annualised SR divided by \(\sqrt{365}\).

      The probabilistic Sharpe ratio (PSR) is the probability that the true SR exceeds a benchmark \(\mathrm{SR}^*\), given the estimate \(\widehat{\mathrm{SR}}\) over \(T\) days and the skewness \(\hat\gamma_3\) and kurtosis \(\hat\gamma_4\) of the daily returns:

      \[\mathrm{PSR}(\mathrm{SR}^*) = \Phi\!\left(\frac{(\widehat{\mathrm{SR}} - \mathrm{SR}^*)\sqrt{T-1}}{\sqrt{1 - \hat\gamma_3 \widehat{\mathrm{SR}} + \frac{\hat\gamma_4 - 1}{4}\widehat{\mathrm{SR}}^2}}\right)\]
      (8.8)

      Heavy tails and negative skewness widen the denominator and lower the PSR. The deflated Sharpe ratio (DSR) of Bailey & López de Prado (2014) is \(\mathrm{PSR}(\mathrm{SR}_0)\): the benchmark is no longer zero but the best Sharpe ratio expected by chance from formula (8.7). The usual threshold is 0.95: a DSR above it means a probability of at least 95% that the true SR of the winner exceeds \(\mathrm{SR}_0\).

      A second approach, the reality check of White (2000), simulates the distribution of the best SR when no rule has skill, by resampling the returns. The stationary bootstrap resamples blocks of consecutive days of random length, for example with a mean of 20 days, and so keeps the dependence between days, including volatility clustering. A simple random permutation of the days is easier to explain but destroys volatility clustering, so it gives only a rougher illustration.

      An example of such a search has 810 rules: 560 moving-average crossovers MA(\(s\),\(l\)), which are long when the \(s\)-day average of the log price is above the \(l\)-day average, and 250 momentum rules MOM(\(k\)), which are long when the return over the last \(k\) days is positive. The rules are selected on 2018–2022 and evaluated on 2023 – 23 October 2026.

      In brief

      • Among \(N\) rules with no skill, the best one still has a high Sharpe ratio. For \(N\) independent rules with true SR \(= 0\), the expected maximum is approximately (SR in daily units here, \(= \) annualised SR\(/\sqrt{365}\))
        \[ \mathrm{SR}_0 = \sqrt{V[\widehat{\mathrm{SR}}_n]}\left[(1-\gamma)\,\Phi^{-1}\!\left(1 - \tfrac1N\right) + \gamma\, \Phi^{-1}\!\left(1 - \tfrac{1}{N e}\right)\right], \quad \gamma \approx 0.5772 \]
      • Probabilistic Sharpe ratio (PSR): probability that the true SR exceeds a benchmark \(\mathrm{SR}^*\), allowing for skewness \(\hat\gamma_3\) and kurtosis \(\hat\gamma_4\) of daily returns over \(T\) days:
        \[ \mathrm{PSR}(\mathrm{SR}^*) = \Phi\!\left(\frac{(\widehat{\mathrm{SR}} - \mathrm{SR}^*)\sqrt{T-1}}{\sqrt{1 - \hat\gamma_3 \widehat{\mathrm{SR}} + \frac{\hat\gamma_4 - 1}{4}\widehat{\mathrm{SR}}^2}}\right) \]
      • Deflated Sharpe ratio (DSR) \(= \mathrm{PSR}(\mathrm{SR}_0)\): the benchmark is the best Sharpe ratio expected by chance (Bailey & López de Prado (2014))
      • White's (2000) reality check: distribution of the best SR when no rule has skill, by resampling the returns
        • stationary bootstrap: resamples blocks of consecutive days (for example, mean length 20 days) and keeps volatility clustering
        • a simple permutation of the days destroys volatility clustering: a rougher illustration
      • Example of a search: 810 rules (560 moving-average crossovers MA(\(s\),\(l\)) and 250 momentum rules MOM(\(k\))), selected on 2018–2022, evaluated on 2023 – 23 October 2026

      810 Trading Rules

      Slide 15 in the PDF
        Question for the room The ▶ button raises the number of independent trials N in the deflated Sharpe ratio from 2 to 810. Up to which N is the winner MOM(2) significant at 95% (DSR above 0.95)?
        Answer

        Only for N = 2 (DSR 0.953). The benchmark SR₀, the best annualised Sharpe ratio expected by chance, rises from 0.09 (N = 2) to 0.22 (N = 5), 0.29 (N = 10), 0.46 (N = 100) and 0.58 (N = 810); the DSR falls from 0.95 to 0.92, 0.89, 0.80 and 0.72. The PSR against zero, 0.97, ignores the search. The rules are strongly correlated, so the effective N is below 810, but even 5 independent trials leave the winner below 0.95. Out of sample (2023–2026) MOM(2) has a Sharpe ratio of 0.45 against 0.90 for buy-and-hold.

        The left panel plots, for each of the 810 rules, its Sharpe ratio in the selection period 2018–2022 (horizontal axis) against its Sharpe ratio in the evaluation period 2023–2026 (vertical axis). The dashed line is buy-and-hold in 2023–2026, and the red point is the best rule in the selection period. The right panel is the histogram of the best SR among the 810 rules in 200 permutations of the returns, with the red line at the SR of the best rule on the actual data.

        The best rule in 2018–2022 is MOM(2), long when the return of the last two days is positive, with SR \(= 0.84\) against 0.04 for buy-and-hold. In 2023–2026 its SR falls to 0.45, against 0.90 for buy-and-hold, and it ranks 779th of 810. The cloud of points shows why: the rank correlation between in-sample and out-of-sample SR is only 0.15, and only 12% of the rules beat buy-and-hold after 2022.

        The corrections reach the same conclusion before the evaluation period is seen. In the permutations, the best of 810 rules reaches an SR of 0.55 on average and 0.93 at the 95th percentile, so the winner's 0.84 has \(p = 0.095\); the reality check with the stationary bootstrap gives \(p = 0.31\). The PSR against zero is 0.97, but the DSR with \(N = 810\) is 0.72: once the search is counted, the winner is not significant.

        The 810 rules are strongly correlated, since neighbouring windows give similar signals, so the effective number of independent trials is smaller than 810. With \(N = 100\) the DSR is 0.80 and with \(N = 10\) it is 0.89, still below 0.95.

        In brief

        • Best rule in 2018–2022: MOM(2), SR \(= 0.84\) against 0.04 for buy-and-hold. In 2023–2026: SR \(= 0.45\) against 0.90, rank 779 of 810
        • Permutation illustration: the best of 810 rules reaches 0.55 on average and 0.93 at the 95th percentile: \(p = 0.095\) for the winner. Reality check (stationary bootstrap): \(p = 0.31\)
        • PSR against zero \(= 0.97\), but DSR \(= 0.72\) with \(N = 810\): once the search is counted, the winner is not significant
        • The 810 rules are strongly correlated, so the effective number of trials is smaller: DSR 0.80 for \(N = 100\), 0.89 for \(N = 10\); still below 0.95
        • Rank correlation of in-sample and out-of-sample SR: 0.15; only 12% of the rules beat buy-and-hold after 2022
        • Source: own computation on EODHD daily Bitcoin prices, 2018 – 23 October 2026
        BA_L8_snoop

        Inflation of the Best Sharpe Ratio by Search

        Slide 16 in the PDF

        The deflated Sharpe ratio depends on the number of rules tried, \(N\). The chart shows how. The left panel plots the expected best annualised Sharpe ratio of \(N\) rules with no skill, \(\mathrm{SR}_0\) from formula (8.7), using the spread of the 810 in-sample Sharpe ratios of the lecture's rules; the red dashed line is the Sharpe ratio of the winner, 0.84, and the dotted line marks \(N = 810\).

        The expected best Sharpe ratio grows quickly with \(N\): 0.09 for two rules, 0.29 for ten, 0.46 for a hundred and 0.58 for 810. The winner's 0.84 stays above \(\mathrm{SR}_0\) at every \(N\) shown, but the margin shrinks as the search grows.

        The right panel turns the margin into the DSR, the probability that the true Sharpe ratio of the winner exceeds \(\mathrm{SR}_0\). It falls below the threshold of 0.95 already from \(N = 3\) rules, and reaches 0.72 at \(N = 810\). Even if the 810 correlated rules count as far fewer independent trials, the winner is not significant once the search is taken into account.

        In brief

        Inflation of the Best Sharpe Ratio by Search
        • Left: expected best annualised SR of \(N\) rules with no skill, \(\mathrm{SR}_0\), using the spread of the 810 in-sample SRs: 0.09 for \(N = 2\), 0.29 for 10, 0.46 for 100, 0.58 for 810
        • The winner's 0.84 is above \(\mathrm{SR}_0\) at every \(N\), but the margin shrinks as the search grows
        • Right: the deflated Sharpe ratio of the winner falls below 0.95 from \(N = 3\) rules on, and is 0.72 at \(N = 810\)
        • Source: own computation on EODHD daily Bitcoin prices, 2018 – 23 October 2026
        BA_L8_dsr

        Worked Example: Deflating the Best Sharpe Ratio

        Slide 17 in the PDF

        The worked example computes the DSR of the winner step by step. The winning rule is MOM(2), long when the price is above its level two days earlier. Over the \(T = 1{,}826\) days of 2018–2022 its daily Sharpe ratio is \(\widehat{\mathrm{SR}} = 0.0438\), which is \(0.0438 \times \sqrt{365} = 0.84\) annualised; the skewness of its daily returns is \(\hat\gamma_3 = 0.58\) and their kurtosis \(\hat\gamma_4 = 11.46\).

        The first step measures how much the Sharpe ratios of the 810 rules vary: the standard deviation of their daily values is 0.00953. The second step takes the two Normal quantiles of formula (8.7) for \(N = 810\): \(\Phi^{-1}(1 - 1/810) = 3.027\) and \(\Phi^{-1}(1 - 1/(810e)) = 3.317\). The third step combines them with Euler's constant: \(\mathrm{SR}_0 = 0.00953 \times (0.423 \times 3.027 + 0.577 \times 3.317) = 0.00953 \times 3.195 = 0.0304\), or 0.58 annualised.

        The fourth step computes the denominator of the PSR, which corrects for skewness and heavy tails: \(\sqrt{1 - 0.58 \times 0.0438 + \frac{11.46 - 1}{4} \times 0.0438^2} = \sqrt{0.980} = 0.990\). The fifth step forms the statistic \((0.0438 - 0.0304) \times \sqrt{1{,}825}/0.990 = 0.578\) and the DSR \(= \Phi(0.578) = 0.72\).

        The same steps with the benchmark \(\mathrm{SR}^* = 0\) give \(0.0438 \times \sqrt{1{,}825}/0.990 = 1.89\) and a PSR of \(\Phi(1.89) = 0.97\). Against zero the winner looks significant; against the best result expected from a search of 810 rules it does not. The only difference between the two verdicts is that the second counts the search.

        In brief

        • Winner MOM(2), 2018–2022: \(T = 1\,826\) days; daily \(\widehat{\mathrm{SR}} = 0.0438\) (\(\times\sqrt{365} = 0.84\)); skewness \(\hat\gamma_3 = 0.58\), kurtosis \(\hat\gamma_4 = 11.46\)
        • Step 1, spread of the 810 daily SRs: \(\sqrt{V[\widehat{\mathrm{SR}}_n]} = 0.00953\)
        • Step 2, Normal quantiles for \(N = 810\): \(\Phi^{-1}(1 - 1/810) = 3.027\), \(\;\Phi^{-1}(1 - 1/(810e)) = 3.317\)
        • Step 3, benchmark: \(\mathrm{SR}_0 = 0.00953 \times (0.423 \times 3.027 + 0.577 \times 3.317) = 0.00953 \times 3.195 = 0.0304\) (annualised 0.58)
        • Step 4, denominator: \(\sqrt{1 - 0.58 \times 0.0438 + \tfrac{11.46 - 1}{4} \times 0.0438^2} = \sqrt{0.980} = 0.990\)
        • Step 5, statistic: \((0.0438 - 0.0304)\sqrt{1\,825}/0.990 = 0.578\); DSR \(= \Phi(0.578) = 0.72\)
        • Same steps with \(\mathrm{SR}^* = 0\): \(0.0438 \times \sqrt{1\,825}/0.990 = 1.89\), PSR \(= \Phi(1.89) = 0.97\). The search alone moves the verdict from “significant” to “not significant”
        • Source: own computation on EODHD daily Bitcoin prices, 2018 – 23 October 2026

        Section summary

        The best of 810 trading rules in 2018–2022 ranked 779th in the following years. The deflated Sharpe ratio and the reality check would have flagged the winner as a likely product of the search, and reporting the number of rules tried is the minimum needed to judge any back-test.

        Risk Forecasts: Classical, ML and Pretrained

        The direction of tomorrow's return is hard to forecast; its size is easier, because volatility clusters (Lecture 4). The section compares six forecasts of Bitcoin's one-day Value-at-Risk, three classical, two from machine learning and one from a pretrained model, with two scoring rules that can rank them differently.

        Six Value-at-Risk Models

        Slide 18 in the PDF

        Let \(q_t\) be the \(\alpha\)-quantile of the return \(r_t\) given information up to day \(t-1\); for small \(\alpha\) it is negative. The one-day Value-at-Risk (VaR) is \(\mathrm{VaR}_t = -q_t\), a positive loss (Lecture 4). The lecture uses \(\alpha = 1\%\) and \(5\%\), written VaR 1% and VaR 5%, with returns in percent.

        The three classical models are those of Lecture 4. Historical simulation (HS) takes the empirical \(\alpha\)-quantile of the last 500 returns. The exponentially weighted moving average with a Normal quantile (EWMA-N) updates the variance as \(\sigma_t^2 = 0.94\,\sigma_{t-1}^2 + 0.06\, r_{t-1}^2\) and sets \(q_t = \Phi^{-1}(\alpha)\sigma_t\). The GARCH(1,1)-t model, a generalised autoregressive conditional heteroskedasticity model with Student-\(t\) errors, gives \(q_t = \mu + \sigma_t\, t_\nu^{-1}(\alpha)\sqrt{(\nu-2)/\nu}\) and is refitted every 21 days.

        The two machine-learning models forecast the quantile directly. The linear quantile regression (QReg) of Koenker & Bassett (1978) uses heterogeneous autoregressive (HAR) terms, volatilities over different horizons:

        \[q_t = \beta_0 + \beta_1 |r_{t-1}| + \beta_2 \mathrm{RV}^{(5)}_{t-1} + \beta_3 \mathrm{RV}^{(22)}_{t-1} + \beta_4 \mathrm{RV}^{(66)}_{t-1} + \beta_5 \min(r_{t-1},0)\]
        (8.9)

        where \(\mathrm{RV}^{(k)}\) is the root mean square return over the last \(k\) days. The gradient-boosted quantile trees (QGBM) fit 300 trees that minimise the quantile loss on 9 features (returns, volatilities and the volume change), on a rolling window of 1,500 days.

        The pretrained model is Chronos-2 of Ansari et al. (2025), a transformer neural network trained on a large collection of real and synthetic time series. It returns quantiles from 1% to 99% directly and is used zero-shot: no parameter is fitted to Bitcoin, and its only input is the last 512 daily returns. Forecasts start on 1 June 2022, and the test period runs from 1 January 2024 to 23 October 2026, \(T = 1{,}027\) days.

        In brief

        • \(q_t\): the \(\alpha\)-quantile of the return \(r_t\) given information to \(t-1\) (negative); one-day Value-at-Risk \(\mathrm{VaR}_t = -q_t\), a positive loss (Lecture 4); \(\alpha = 1\%\) and \(5\%\); returns in %
        • Classical (Lecture 4):
          • historical simulation (HS): empirical \(\alpha\)-quantile of the last 500 returns
          • EWMA-N: \(\sigma_t^2 = 0.94\,\sigma_{t-1}^2 + 0.06\, r_{t-1}^2\), \(q_t = \Phi^{-1}(\alpha)\sigma_t\)
          • GARCH(1,1)-t: \(q_t = \mu + \sigma_t\, t_\nu^{-1}(\alpha)\sqrt{(\nu-2)/\nu}\), refitted every 21 days
        • Machine learning:
          • linear quantile regression (QReg, Koenker & Bassett (1978)) on heterogeneous autoregressive (HAR) terms: \(\; q_t = \beta_0 + \beta_1 |r_{t-1}| + \beta_2 \mathrm{RV}^{(5)}_{t-1} + \beta_3 \mathrm{RV}^{(22)}_{t-1} + \beta_4 \mathrm{RV}^{(66)}_{t-1} + \beta_5 \min(r_{t-1},0)\), with \(\mathrm{RV}^{(k)}\) the \(k\)-day root mean square return
          • gradient-boosted quantile trees (QGBM): 300 trees minimising the quantile loss on 9 features (returns, volatilities, volume change); rolling 1 500 days
        • Pretrained: Chronos-2 (Ansari et al. (2025)), a transformer trained on a large corpus of real and synthetic series; it returns quantiles from 1% to 99% directly. Zero-shot: no parameter is fitted to Bitcoin; input = the last 512 daily returns
        • Forecasts from 1 June 2022; test 1 January 2024 – 23 October 2026 (\(T = 1\,027\) days)

        How to Score a Quantile Forecast

        Slide 19 in the PDF

        The usual check of a VaR forecast counts breaches, the days on which the loss exceeds the VaR: \(h_t = \mathbb{1}\{r_t < q_t\}\). The breach rate \(\bar h\) should be close to \(\alpha\). The Kupiec likelihood ratio (LR) test asks whether \(\bar h = \alpha\), and the Christoffersen test whether breaches occur independently over time rather than in clusters (Lecture 4).

        The breach rate checks only the average level of the forecast. The quantile score (QS), also called pinball loss, scores every day:

        \[\mathrm{QS}_\alpha(q, r) = (r - q)\left(\alpha - \mathbb{1}\{r < q\}\right) \ge 0\]
        (8.10)

        On a day without a breach the loss is \(\alpha\) times the distance between the return and the quantile; on a breach day it is \(1-\alpha\) times that distance, so a breach costs 99 times more per unit for \(\alpha = 1\%\). The QS is strictly consistent for the quantile: its expected value is minimised only by the true conditional quantile. The breach rate is not. A forecast equal to \(-\infty\) on 99% of days and \(+\infty\) on the remaining 1% has a perfect breach rate and is useless.

        Two models \(A\) and \(B\) are compared with the Diebold–Mariano (DM) test on the daily loss difference \(d_t = \mathrm{QS}^{A}_t - \mathrm{QS}^{B}_t\):

        \[\mathrm{DM} = \frac{\bar d}{\sqrt{\hat\Omega_d / T}} \;\overset{a}{\sim}\; N(0,1)\]
        (8.11)

        where \(\hat\Omega_d\) is the long-run variance of \(d_t\), estimated with the heteroskedasticity- and autocorrelation-consistent (HAC) method of Newey and West. In short, the breach rate answers "is the level right on average?", while the mean QS answers "how large is the average loss?", with a loss whose expectation is smallest for the true quantile.

        In brief

        • Breach indicator \(h_t = \mathbb{1}\{r_t < q_t\}\); breach rate \(\bar h\) should be close to \(\alpha\)
        • Kupiec likelihood ratio (LR) test of \(\bar h = \alpha\); Christoffersen test of independent breaches (Lecture 4)
        • Quantile score (QS), also called pinball loss:
          \[ \mathrm{QS}_\alpha(q, r) = (r - q)\left(\alpha - \mathbb{1}\{r < q\}\right) \ge 0 \]
        • The QS is strictly consistent for the quantile: its expectation is minimised only by the true conditional quantile. The breach rate is not: the forecast \(q_t = -\infty\) on 99% of days and \(+\infty\) on 1% has a perfect breach rate and is useless
        • Diebold–Mariano (DM) test on the loss difference \(d_t = \mathrm{QS}^{A}_t - \mathrm{QS}^{B}_t\):
          \[ \mathrm{DM} = \frac{\bar d}{\sqrt{\hat\Omega_d / T}} \;\overset{a}{\sim}\; N(0,1), \qquad \hat\Omega_d = \text{HAC (Newey--West) long-run variance} \]
        • Breach rate answers “is the level right on average?”; the mean QS answers “how large is the average loss?”, a loss whose expectation is smallest for the true quantile

        The Quantile Score: a Graphical Illustration

        Slide 20 in the PDF

        The left panel draws the quantile score of a single day for \(\alpha = 1\%\) as a function of the forecast error \(r - q\). When the return falls below the forecast quantile (a breach, \(r - q < 0\)), the loss grows with slope 0.99 per percentage point; when the return stays above it, the loss grows with slope 0.01. A breach of 5 points therefore costs 4.95, while a day 5 points above the quantile costs 0.05. The asymmetry reflects the level: the 1% quantile should be breached on only one day in a hundred.

        The right panel shows why this score rewards the true quantile. For each constant forecast \(q\) between \(-16\%\) and \(-1\%\) it computes the mean score over the 1,027 test days, 1 January 2024 – 23 October 2026. The curve is lowest, 0.0827, at the empirical 1% quantile of the test returns, \(-6.48\%\). A constant \(q = -5\%\), too close to zero, gives 2.6% breaches and a mean score of 0.0912; a constant \(q = -10\%\), too far out, gives 0.1% breaches and 0.1057.

        The horizontal line is the mean score of GARCH-t, 0.0803. It lies below the whole curve: a forecast that moves with the volatility of the market beats every fixed quantile, even the one chosen with hindsight. This is the gain from conditioning on information, measured in the same units as the comparisons between models.

        In brief

        The Quantile Score: a Graphical Illustration
        • Left: the loss of one day. A breach costs \(0.99\) per percentage point below \(q\); a day above \(q\) costs only \(0.01\) per point
        • Right: mean QS of a constant 1% quantile over the 1 027 test days. The minimum (0.0827) is at the empirical 1% quantile, \(-6.48\%\): the score rewards the true quantile
        • A constant \(q = -5\%\) (2.6% breaches) scores 0.0912; \(q = -10\%\) (0.1% breaches) 0.1057
        • GARCH-t scores 0.0803, below every constant: a forecast that follows volatility beats the best fixed one
        • Source: own computation on EODHD daily Bitcoin prices and volumes; test 1 January 2024 – 23 October 2026
        BA_L8_pinball

        Worked Example: the Quantile Score of Two Days

        Slide 21 in the PDF

        Two days of the GARCH-t forecast show how the quantile score works in practice. On day A, 5 March 2024, GARCH-t forecast a 1% quantile of \(q = -10.62\%\), a VaR of 10.62%, and Bitcoin fell 6.86%. The error is \(r - q = -6.86 + 10.62 = 3.76 > 0\), so there is no breach and the indicator is 0. The score is \(3.76 \times 0.01 = 0.0376\): a large loss, but above the forecast, costs little, and the small cost comes from a VaR set further out than needed.

        On day B, 5 February 2026, GARCH-t forecast \(q = -8.92\%\), a VaR of 8.92%, and Bitcoin fell 15.23%. Now \(r - q = -15.23 + 8.92 = -6.32 < 0\), the indicator is 1, and the score is \(-6.32 \times (0.01 - 1) = 6.25\), 166 times the score of day A.

        Over the 1,027 test days the mean score of GARCH-t is 0.0803, a total of 82.5; day B alone contributes 7.6% of it. The mean quantile score is therefore driven by a handful of breach days. With about ten breaches expected at the 1% level, the difference in mean score between two models depends on a few days, which is why the Diebold–Mariano tests of the table rarely reject.

        In brief

        • GARCH-t, 1% quantile, \(\mathrm{QS}_{0.01}(q, r) = (r - q)(0.01 - \mathbb{1}\{r < q\})\), returns in %
        • Day A, 5 March 2024, no breach: \(q = -10.62\) (VaR 10.62%), \(r = -6.86\)
          • \(r - q = 3.76 > 0\), so \(\mathbb{1} = 0\): \(\;\mathrm{QS} = 3.76 \times 0.01 = 0.0376\)
          • a large loss, but above the forecast: the cost is small and comes from a VaR set too far out
        • Day B, 5 February 2026, breach: \(q = -8.92\) (VaR 8.92%), \(r = -15.23\)
          • \(r - q = -6.32 < 0\), so \(\mathbb{1} = 1\): \(\;\mathrm{QS} = -6.32 \times (0.01 - 1) = 6.25\)
          • one day, 166 times the loss of day A
        • Mean over the 1 027 test days: 0.0803, a total of 82.5; day B alone is 7.6% of it
        • The mean QS is driven by a few breaches; with about 10 breaches expected, differences between models are noisy
        • Source: own computation on EODHD daily Bitcoin prices and volumes; test 1 January 2024 – 23 October 2026

        Bitcoin VaR, 2024 – October 2026

        Slide 22 in the PDF

        The table reports, for each of the six models, the breach rate, the Kupiec and Christoffersen p-values, the mean QS and the DM p-value against GARCH-t, at \(\alpha = 1\%\) (left) and \(5\%\) (right). At 1%, 1,027 test days lead to 10.3 expected breaches.

        At VaR 1%, all models except EWMA-N pass the coverage tests. EWMA-N breaches on 2.14% of days, more than twice the target, and the Kupiec test rejects it (p 0.001): the Normal quantile is too thin for Bitcoin's tails. The breach rate closest to 1% belongs to QGBM (1.07%), but QGBM does not have the lowest QS. GARCH-t has the lowest mean QS in this test, 0.0803, with 7 breaches (0.68%) against 10.3 expected: it is slightly conservative.

        The DM tests show that the differences in QS are small relative to their noise. At 1%, no difference in expected QS between GARCH-t and any other model is detected; all DM p-values are 0.18 or higher. For example, Chronos-2 minus GARCH-t is 0.0025, with a 95% confidence interval of \([-0.0053, 0.0102]\). Not rejecting equality does not show that the models are equally good: with about 10 expected breaches, the test has little power.

        At VaR 5% there are about five times as many breaches, which gives the tests more power. All models pass the Kupiec test at 5%, and QGBM is the weakest by QS, with a DM p-value of 0.07 against GARCH-t.

        In brief

        \(\alpha = 1\%\) (10.3 breaches expected)\(\alpha = 5\%\)
        modelbreach %Kupiec pChrist. pQSDM pbreach %Kupiec pQSDM p
        HS1.170.600.590.08420.455.360.610.27820.57
        EWMA-N2.140.0010.330.08800.185.260.710.27850.18
        GARCH-t0.680.280.760.0803–5.650.350.2748–
        QReg0.880.680.690.08220.713.890.090.27550.92
        QGBM1.070.820.630.08480.366.130.110.28750.07
        Chronos-21.270.410.560.08270.545.160.810.27730.71
        • DM p: two-sided test of equal mean QS against GARCH-t
        • All models except EWMA-N pass the coverage tests. The best breach rate (QGBM, 1.07%) does not have the lowest QS; GARCH-t has the lowest mean realised QS in this test, with 7 breaches against 10.3 expected
        • At 1%, no difference in expected QS from GARCH-t is detected: Chronos-2 minus GARCH-t \(= 0.0025\), 95% interval \([-0.0053, 0.0102]\). Not rejecting equality does not show that the models are equally good
        • The 5% level gives more breaches and more power; QGBM is the weakest there (p 0.07)
        • Source: own computation on EODHD daily Bitcoin prices and volumes; test 1 January 2024 – 23 October 2026

        Breaches Day by Day

        Slide 23 in the PDF

        The breach rates of the table are totals over the whole test period. The chart shows how they accumulate. Each step line counts the breaches of one model's 1% VaR from 1 January 2024 to 23 October 2026; the red dashed line is the expected count, \(0.01 \times\) the number of days, which reaches 10.3 at the end, and the shaded band is the 95% range of a binomial count on each day, 5 to 17 breaches at the end.

        EWMA-N, with its Normal quantile, leaves the band on 5 August 2024 and is above it on 553 days from then on; it ends with 22 breaches, the failure of the Kupiec test in the table. Historical simulation and Chronos-2 rise above the band on some days in 2024, HS between April and August and Chronos-2 between April and October: Chronos-2 had 7 breaches in 2024, against 3.7 expected. Both then return inside the band.

        GARCH-t, QReg and QGBM stay inside the band throughout, with 7, 9 and 11 breaches. The chart shows that a total breach count within the band can hide a period of too many breaches, and that a backtest over a short window can give a different verdict from the same backtest over the full period.

        In brief

        Breaches Day by Day
        • Cumulative breaches of each 1% VaR, 1 January 2024 – 23 October 2026; dashed: expected \(0.01 \times\) days (10.3 at the end); band: 95% binomial range for each day (5 to 17 at the end)
        • EWMA-N is above the band on 553 days from 5 August 2024 on and ends with 22 breaches: the Kupiec failure of the table
        • HS (April – August 2024) and Chronos-2 (April – October 2024) are above the band on some days; Chronos-2 has 7 breaches in 2024 against 3.7 expected, then returns inside the band
        • GARCH-t, QReg and QGBM stay inside the band throughout (7, 9 and 11 breaches)
        • Source: own computation on EODHD daily Bitcoin prices and volumes; test 1 January 2024 – 23 October 2026
        BA_L8_breaches

        Model Rankings by Breach Rate and by Quantile Score

        Slide 24 in the PDF
          Question for the room The ▶ button extends the evaluation period of the VaR 1% month by month, from January 2024 to October 2026. Which ranking of the six models is the more stable?
          Answer

          The ranking by mean quantile score. From May 2024 on, GARCH-t has the lowest mean QS at every month end; QReg is lowest in the first four months. The model closest to the 1% breach target changes 10 times, among five models (HS, GARCH-t, EWMA-N, QReg, QGBM). With 10.3 breaches expected in 1,027 days, one breach more or less moves a breach rate by 0.1 percentage point, while the QS uses the size of the miss on every day.

          The left panel shows the breach rate of each model at VaR 1%, with the target of 1% as a dashed line; the right panel shows the mean quantile score, where lower is better. The two rankings differ. Closest to the target are QGBM, HS and Chronos-2; lowest in QS are GARCH-t, QReg and Chronos-2.

          Note that the vertical axis of the right panel starts at 0.0725, which makes the differences in QS look larger than they are; the DM tests above found none of the differences at VaR 1% significant. The disagreement between the panels has a simple cause. A forecast can hit the right number of breaches on average while being too high on calm days and too low on volatile days. The breach rate does not see this; the QS penalises both errors.

          In brief

          • Closest to target: QGBM, HS, Chronos-2. Lowest QS: GARCH-t, QReg, Chronos-2
          • A forecast can hit the right breach count while being too high on calm days and too low on volatile days; the QS penalises both
          • Source: own computation on EODHD daily Bitcoin prices and volumes; test 1 January 2024 – 23 October 2026
          BA_L8_var_models

          VaR Paths of the Tree Model and GARCH-t

          Slide 25 in the PDF
            Question for the room On 14 January 2024 the VaR 1% of QGBM, the gradient-boosted quantile trees, jumps from 7.3% to 35.4%. The ▶ button moves day by day through January 2024. What is the QGBM VaR 1% on the next day, 15 January?
            Answer

            8.1%: the jump lasts one day. On 14 January the trees received a −7.9% return on 12 January followed by a 52% fall in volume on 13 January, a combination met in training only in crash periods; on 15 January the inputs (return −2.5%, volume −15%) fall in an ordinary leaf again. GARCH-t moves smoothly, from 9.9% (13 January) to 9.4% and 9.2%; a variance recursion cannot jump like this, and a tree forecast has no memory from one day to the next.

            The chart shows the 1% return quantiles \(q_t\) of three models from January 2024 to October 2026: GARCH-t (dark blue), QGBM (green) and Chronos-2 (purple), with the daily Bitcoin log returns as orange bars. The VaR is \(-q_t\), so a line further down means a higher VaR. Red points mark breaches of the Chronos-2 quantile, and the dotted vertical line marks the release of Chronos-2 on 20 October 2025.

            The most striking feature is a single spike of the tree model. On 14 January 2024, the VaR 1% of QGBM jumps from 7.3% to 35.4%, while that of GARCH-t moves from 9.9% to 9.4%. The input behind the jump was a return of \(-7.9\%\) on 12 January, the day after the launch of the spot Bitcoin exchange-traded funds (ETFs), followed by a 52% fall in volume on 13 January: a combination that the trees had met only in crash periods.

            Trees do not extrapolate smoothly. An unusual input lands in an extreme leaf of a tree, and the forecast jumps to the value learnt from the few crash days in that leaf. A risk manager using such a model needs bounds on the forecast or a fallback rule, such as a classical model that takes over when the ML forecast moves too far in one day.

            In brief

            • Chart: 1% return quantiles \(q_t\); the VaR is \(-q_t\)
            • On 14 January 2024 the tree model's 1% VaR jumps from 7.3% to 35.4%; GARCH-t moves from 9.9% to 9.4%
            • The input: a \(-7.9\%\) return on 12 January, the day after the spot Bitcoin exchange-traded fund (ETF) launch, then a 52% fall in volume on 13 January, a combination the trees met only in crash periods
            • Trees do not extrapolate smoothly: an unusual input lands in an extreme leaf. A risk manager needs bounds or a fallback rule
            • Source: own computation on EODHD daily Bitcoin prices and volumes; test 1 January 2024 – 23 October 2026
            BA_L8_var_paths

            Pretrained Models and Look-Ahead

            Slide 26 in the PDF

            A pretrained model raises a problem that a model fitted by the analyst does not: it may have seen the test period during its training. Chronos-2 was released in October 2025, and its training corpus may include Bitcoin prices for 2024–2025. Only forecasts made after the release are clean; earlier forecasts are a retrospective back-test that may benefit from look-ahead, the use of information not available at the forecast date.

            A second problem is reproducibility. A pretrained model is defined by its weights, and the weights published under a model name can be replaced. Loading the model by name without a fixed version, called a revision, can bring different weights later, and the results would change without notice. A back-test of a pretrained model must therefore report the exact version it used.

            A simple test for look-ahead splits the test period at the release date, here 20 October 2025, and comparing the mean QS at VaR 1% before (659 days) and after (368 days). Chronos-2 ranks third in both periods, with a QS of 0.082 before and 0.084 after. The best model changes, from GARCH-t (0.077) before to QReg (0.077) after, and EWMA-N breaches on 2.17% of days after the release. Chronos-2 shows no sign of an advantage from leaked data, but 368 days with about 4 expected breaches cannot prove that there was none.

            The same problem is stronger for large language models (LLMs), which are trained on large parts of the internet, including news and prices: a back-test of an LLM must use dates after its training cut-off. Other pretrained models have other limits: TimesFM 2.5 returns only deciles, from 10% to 90%, so it cannot give a VaR 1% without extrapolating beyond its output.

            In brief

            • A pretrained model may have seen the test period during training. Chronos-2 was released in October 2025; its training corpus may include Bitcoin prices for 2024–2025. Only forecasts after the release are clean; earlier ones are a retrospective back-test
            • Reproducibility: a model name without a fixed version (revision) can load other weights later; a back-test must report the exact version
            • A test for look-ahead: score the same model before and after its release date (20 October 2025); 1% VaR, mean QS:
            HSEWMA-NGARCH-tQRegQGBMChronos-2
            before release (659 days): QS0.0810.0830.0770.0850.0860.082
            after release (368 days): QS0.0890.0970.0870.0770.0830.084
            after release: breach %0.822.170.541.091.091.09
            • Chronos-2 ranks third in both periods: no sign of an advantage from leaked data, but 368 days and 4 expected breaches cannot prove its absence
            • Large language models (LLMs): same problem, stronger; a back-test of an LLM must use dates after its training cut-off
            • Other pretrained models: TimesFM 2.5 returns only deciles (10%–90%), so it cannot give a 1% VaR without extrapolation

            Rolling Quantile Scores Relative to GARCH-t

            Slide 27 in the PDF
              Question for the room The ▶ button lengthens the rolling window from 60 to 720 days. What happens to the share of windows in which EWMA-N, the Normal-quantile model, has a lower mean 1% quantile score than GARCH-t?
              Answer

              It falls from 59% (60-day windows) to 14% (365 days) and to 0% from 480 days on. A 1% VaR is breached on about one day in a hundred, so most short windows contain no breach of either model; on such days a lower VaR scores better, and EWMA-N, with its thin Normal tail, usually has the lower VaR. Long windows contain the breaches, where its thin tail costs most. Short windows therefore reward the wrong model; the other models show the same pattern: Chronos-2 is ahead in 60% of the 60-day windows and in 13–22% of the windows of 300 days or more.

              The look-ahead check split the test period at one date. The chart follows the comparison continuously: for each model it plots the mean 1% quantile score over the last 365 days minus that of GARCH-t, in units of \(10^{-3}\). Below zero, the model has had the lower mean score over the past year; the first window ends on 30 December 2024.

              In the last window, to 23 October 2026, three models are ahead of GARCH-t: QReg by 9.9, QGBM by 3.4 and Chronos-2 by 2.7 units. EWMA-N (10.3 units) and HS (2.6) are behind. Over all windows, however, Chronos-2 is ahead in only 16% of them and QReg in 38%. The dotted line marks the release of Chronos-2, 20 October 2025.

              The jumps in the lines are single breach days entering or leaving the 365-day window. A breach of a few percentage points changes a model's mean score over a year by \(0.01\) or more, the same order as the differences between models, against a mean score of about 0.08. Rankings over one year are therefore unstable, and a ranking reported for one period should not be expected to hold in the next.

              In brief

              • Mean 1% QS of each model minus that of GARCH-t over the last 365 days (\(\times 10^{-3}\)); below zero = better than GARCH-t; first window ends 30 December 2024
              • In the last window (to 23 October 2026) QReg (\(-9.9\)), QGBM (\(-3.4\)) and Chronos-2 (\(-2.7\)) are ahead of GARCH-t; EWMA-N (\(+10.3\)) and HS (\(+2.6\)) behind
              • Chronos-2 is ahead in 16% of the windows, QReg in 38%; the jumps are single breach days entering or leaving the window
              • Differences of \(\pm 0.01\) against a mean QS of 0.08 are of the size that one large breach creates
              • Source: own computation on EODHD daily Bitcoin prices and volumes; test 1 January 2024 – 23 October 2026
              BA_L8_qs_roll

              Section summary

              For Bitcoin's VaR 1% in 2024–2026, the ML and pretrained models do not differ detectably from GARCH-t in quantile score. Breach rate and QS rank the models differently, tree models can produce extreme jumps after unusual inputs, and a pretrained model must be tested on dates after its release, with a fixed version.

              The Limits of ML and AI in Crypto Finance

              The results of the first three sections are not specific to the models chosen. They follow from features of crypto data and of the way models are evaluated. The section collects them in seven limits.

              Limits of Machine-Learning Forecasts in Crypto Markets

              Slide 28 in the PDF

              The first limit is low signal. The daily direction of Bitcoin is close to a coin toss: the base rate is 50.6% and the best out-of-sample accuracy 51.7%. More flexible models add variance to the forecast, not signal. The second is non-stationarity: market regimes change, from the boom of 2020–2021 to the crash of 2022 and the era of the ETFs from 2024, and what is learnt in one regime may not transfer to the next. The falling accuracy by year and the failed correction of historical simulation in the Appendix are two examples.

              The third limit is the small number of tail observations. A 1% quantile estimated from 500 days rests on about 5 observations, so tail models are noisy by construction (Lectures 4 and 6). The fourth is data snooping: testing many models and reporting the best inflates performance, so the number of trials must be reported and the result deflated. The fifth is look-ahead: pretrained models and LLMs may have seen the test period, features must be known at the forecast time, and model selection must use only past data, the problem of the BitMood trading rule in Lecture 7.

              The sixth limit is survivorship: today's list of coins leaves out the coins that failed, so results computed on survivors overstate performance. The seventh is that explanation is not cause: SHAP values and feature importances describe the model, and causal claims need a design such as the event study of Lecture 7.

              ML does have uses in this field when it is evaluated out of sample with the right score: flexible quantile forecasts with no detectable loss against GARCH, early-warning classifiers that combine many features (Lecture 5), and text-based sentiment indices (Lecture 7).

              In brief

              1. Low signal: daily return direction is close to a coin toss (base rate 50.6%, best accuracy 51.7%); flexibility adds variance, not signal
              2. Non-stationarity: regimes change (2020–2021 boom, 2022 crash, 2024 ETF era); what is learnt in one regime may not transfer (accuracy by year, HS correction)
              3. Few tail observations: a 1% quantile from 500 days rests on 5 points; tail models are noisy by construction (Lectures 4 and 6)
              4. Data snooping: testing many models and reporting the best inflates performance; report the number of trials and deflate
              5. Look-ahead: pretrained models and LLMs may have seen the test period; features must be known at the forecast time; selection must use only past data (BitMood, Lecture 7)
              6. Survivorship: today's list of coins excludes the dead ones; results on survivors overstate performance
              7. Explanation \(\neq\) cause: SHAP and feature importances describe the model; causal claims need a design (event study, Lecture 7)
              • What ML does well here: flexible quantile forecasts with no detectable loss against GARCH, early-warning classifiers with many features (Lecture 5), text-based sentiment (Lecture 7), when evaluated out of sample with the right score

              Exam Review

              The section describes the format of the exam and lists, lecture by lecture, the tools and results that students should be able to interpret. It ends with a worked model answer.

              The Exam: Friday 27 November 2026

              Slide 29 in the PDF

              The exam is taken on the course website and lasts 60 minutes; no books, notes or other materials are allowed. The exam counts for 50% of the final grade and the due-diligence project with Prof. Alon for the other 50%; attendance is strongly recommended.

              The exam has 30 multiple-choice questions, one point each: 15 on Crypto Economics and 15 on Blockchain Analytics, mixed together. The last 5 questions are about a short case built on real output from the course: tables, charts and tests. Each student gets a different set of questions on the screen. No calculation is required, and a wrong answer costs nothing.

              To choose between the options of a question on output, check three things. The first is what is measured: the name of the statistic and its null hypothesis or benchmark. The second is what the number says: its direction, size and significance, in the units of the problem. The third is what it does not say: the relevant limitation, such as sample size, regime, multiple testing, look-ahead, or association rather than cause. The interpretation exercises at the end of each lecture, the practice quizzes and the practice exam on the course website are the best preparation.

              In brief

              • On the course website, 60 minutes; no books, notes or other materials
              • Final grade: exam 50%, due-diligence project with Prof. Alon 50%; attendance strongly recommended
              • 30 multiple-choice questions, one point each: 15 on Crypto Economics and 15 on Blockchain Analytics, mixed together
              • The last 5 questions are about a short case built on real output from the course (tables, charts, tests)
              • Each student gets a different set of questions on the screen; no calculation is required and a wrong answer costs nothing
              • To choose between the options of a question on output, check three things:
                1. What is measured: the statistic and its null hypothesis or benchmark
                2. What the number says: direction, size, significance, in the units of the problem
                3. What it does not say: the limitation (sample size, regime, multiple testing, look-ahead, association vs cause)
              • Preparation: the interpretation exercises at the end of each lecture, the practice quizzes and the practice exam on the course website

              Review: Lectures 1–3

              Slide 30 in the PDF

              Lecture 1 treated blockchains as data. The hash rate follows from the published difficulty, \(\hat H = D \cdot 2^{32}/600\), and measures the cost of securing the network. The supply rule caps Bitcoin at 21 million BTC, with the block subsidy halved every 210,000 blocks; after the 2024 halving, new issuance adds 0.82% a year to the supply. The Herfindahl–Hirschman index (HHI), \(\sum s_i^2\) over the market shares \(s_i\), measures concentration; two issuers hold 85% of all stablecoins. Growth was measured with log-linear trends and their standard errors, and every data set was checked before analysis.

              Lecture 2 studied stylised facts, efficiency and bubbles. Volatility is annualised with \(\sqrt{365}\), because crypto trades every day. Heavy tails are detected with the Jarque–Bera test and quantile–quantile plots, and volatility clustering with the Ljung–Box statistic \(Q(m)\) on squared returns \(r_t^2\), compared with the same test on \(r_t\). Weak-form efficiency is tested with the variance ratio \(\mathrm{VR}(k) = \mathrm{Var}(r_t^{(k)})/(k\,\mathrm{Var}(r_t))\) and its robust test. Bubbles were modelled with the log-periodic power law (LPPL), whose critical time \(t_c\) is a fitted parameter, not a forecast, and network value with Metcalfe's law \(V \propto N^b\).

              Lecture 3 built crypto indices. CRIX, the CRypto IndeX, is a market-value index: the sum of the constituents' market values divided by a divisor, which is reset at each rebalancing so that a change of the basket does not make the index jump. An index of today's coins suffers from survivorship bias, and capping the weights below 50% gives a different product, with more Ethereum and less Bitcoin. VCRIX, the volatility index of CRIX, follows the realised volatility over a 30-day window, so a crash raises it when it enters the window and lowers it again when it leaves. A replica or a fund is compared with its index by the tracking error, how much it wanders around the index, and the tracking difference, by how much it lags on average.

              In brief

              • L1 Blockchains as data
                • hash rate from difficulty, \(\hat H = D \cdot 2^{32}/600\): a measure of security cost
                • supply rule: 21 million BTC, halving every 210 000 blocks; issuance inflation 0.82% after 2024
                • concentration: Herfindahl–Hirschman index (HHI) \(= \sum s_i^2\); two issuers hold 85% of stablecoins
                • log-linear trends and their standard errors; data-quality checks before any analysis
              • L2 Stylised facts, efficiency, bubbles
                • annualise with \(\sqrt{365}\); heavy tails: Jarque–Bera, quantile–quantile plots
                • volatility clustering: Ljung–Box \(Q(m)\) on \(r_t^2\) vs on \(r_t\)
                • weak-form efficiency: variance ratio \(\mathrm{VR}(k) = \mathrm{Var}(r_t^{(k)})/(k\,\mathrm{Var}(r_t))\), robust test
                • bubbles: log-periodic power law (LPPL), critical time \(t_c\), fit is not forecast; Metcalfe's law \(V \propto N^b\)
              • L3 Crypto indices: CRIX, VCRIX and index funds
                • market-value index with a divisor (no jump at rebalancing); survivorship bias; capped weights
                • volatility index VCRIX: 30-day window; tracking error and tracking difference of a replica or a fund

              Review: Lectures 4–5

              Slide 31 in the PDF

              Lecture 4 modelled volatility and tail risk. The GARCH(1,1) model, \(\sigma_t^2 = \omega + \alpha r_{t-1}^2 + \beta\sigma_{t-1}^2\), has persistence \(\alpha + \beta\) and a half-life of shocks, and Student-t errors capture heavy tails. Value-at-Risk and expected shortfall (ES) were computed with four methods: Normal, historical simulation, EWMA and GARCH-t. The backtests were the Kupiec test of the breach frequency, the Christoffersen test of independent breaches and the Basel traffic light. An out-of-sample period showed what a short new period can and cannot reveal about a model.

              Lecture 5 studied the depegs of stablecoins. The deviation from the peg is measured in basis points, \(d_t = 10^4(P_t - 1)\), and grouped into breach episodes whose sizes follow power-law tails. Automated market makers (AMM) price with the constant product \(xy = k\) or with the StableSwap curve, and the composition of a pool can give an early signal of stress. Alarms were evaluated at the level of events: detection rate, lead time, false alerts and utility.

              In brief

              • L4 Volatility and tail risk
                • GARCH(1,1): \(\sigma_t^2 = \omega + \alpha r_{t-1}^2 + \beta\sigma_{t-1}^2\); persistence \(\alpha + \beta\), half-life; Student-t errors
                • VaR and ES; four methods (Normal, HS, EWMA, GARCH-t)
                • backtests: Kupiec (frequency), Christoffersen (independence), Basel traffic light
                • out of sample: what a short new period can and cannot show
              • L5 Stablecoins: depegs
                • deviation in basis points \(d_t = 10^4(P_t - 1)\); breach episodes; power-law tails
                • automated market makers (AMM): constant product \(xy = k\) and StableSwap; pool composition as an early signal
                • event-level evaluation of alarms: detection rate, lead time, false alerts, utility

              Review: Lectures 6–8

              Slide 32 in the PDF

              Lecture 6 built the depeg pressure index (DPI). The check loss \(\rho_\tau(u) = u(\tau - \mathbb{1}\{u<0\})\) defines the 5% Quantile Lasso, whose penalty \(\lambda\) is selected by generalised approximate cross-validation (GACV). The DPI is the average selected penalty: a higher \(\lambda\) means stronger tail co-movement, and under pressure fewer but stronger tail links remain. The lecture also separated in-sample fit from skill on held-out data.

              Lecture 7 covered networks, sentiment and asset classes: the correlation distance \(d_{ij} = \sqrt{2(1-\rho_{ij})}\), the minimum spanning tree and the absorption ratio; Diebold–Yilmaz spillovers (total, TO, FROM and NET); event studies with abnormal and cumulative abnormal returns (AR, CAR) and cross-sectional tests; factor analysis and classification of asset classes; and sentiment indices with out-of-sample selection.

              Lecture 8 added walk-forward evaluation; the base rate, Brier score, AUC and PT test; the distinction between in-sample and out-of-sample results; data snooping and the deflated Sharpe ratio; breach rate against quantile score; and look-ahead in pretrained models.

              In brief

              • L6 The depeg pressure index (DPI)
                • check loss \(\rho_\tau(u) = u(\tau - \mathbb{1}\{u<0\})\); 5% Quantile Lasso; penalty \(\lambda\) selected by generalised approximate cross-validation (GACV)
                • DPI = average selected penalty: a higher \(\lambda\) means stronger tail co-movement; under pressure fewer, stronger tail links remain
                • in-sample fit vs held-out skill
              • L7 Networks, sentiment and asset classes
                • correlation distance \(d_{ij} = \sqrt{2(1-\rho_{ij})}\), minimum spanning tree, absorption ratio
                • Diebold–Yilmaz spillovers: total, TO, FROM, NET
                • event study: abnormal and cumulative abnormal returns (AR, CAR), cross-sectional tests
                • factor analysis and classification: cryptocurrencies as an asset class
                • sentiment indices and out-of-sample selection
              • L8 Machine learning, AI and limits
                • walk-forward evaluation; base rate, Brier, AUC, PT test; in-sample vs out-of-sample
                • data snooping and the deflated Sharpe ratio
                • breach rate vs quantile score; look-ahead in pretrained models

              Worked Example: a Model Answer

              Slide 33 in the PDF

              The example applies the three-part answer to a question on the VaR results: "Bitcoin VaR 1%, 2024 – October 2026: QGBM has a breach rate of 1.07%, GARCH-t of 0.68%; GARCH-t has the lower mean QS (DM p \(= 0.36\)). Which model is better?"

              What is measured. The breach rate checks the average level of the VaR 1%. The mean QS is an average loss whose expectation is smallest for the true quantile. The DM test asks whether the difference in mean QS between the two models is zero.

              What the numbers say. QGBM is closer to 1% on average. GARCH-t is slightly conservative, with fewer breaches than expected, but has the lowest mean realised QS in this test. The two criteria disagree because a model can hit 1% on average by being too high on calm days and too low on volatile days, and only the QS penalises that.

              What it does not say. The numbers do not show that GARCH-t is better. The difference in QS, QGBM minus GARCH-t, is 0.0045, with a 95% interval of \([-0.0051, 0.0141]\) and DM p \(= 0.36\): no difference is detected, and 1,027 days hold only about 10 expected breaches.

              In brief

              • Question: “Bitcoin 1% VaR, 2024 – October 2026: QGBM has a breach rate of 1.07%, GARCH-t of 0.68%; GARCH-t has the lower mean QS (DM p \(= 0.36\)). Which model is better?”
              • What is measured: the breach rate checks the average level of the 1% VaR; the mean QS is an average loss whose expectation is smallest for the true quantile; the DM test asks whether the mean QS difference is zero
              • What the numbers say: QGBM is closer to 1% on average; GARCH-t is slightly conservative but has the lowest mean realised QS in this test
              • Why they disagree: a model can hit 1% on average by being too high on calm days and too low on volatile days; only the QS penalises that
              • What it does not say: that GARCH-t is better. QGBM minus GARCH-t \(= 0.0045\), 95% interval \([-0.0051, 0.0141]\), DM p \(= 0.36\): no difference detected; 1 027 days hold only about 10 expected breaches

              Summary

              Machine learning does not predict the direction of Bitcoin in a useful way. Out-of-sample accuracy is 51.3–51.7% against a base rate of 50.6%, the Brier scores are worse than a constant estimated from past data, and XGBoost classifies 78% of training days correctly but only 51% of new ones. Among 810 trading rules, the best one in 2018–2022 ranked 779th afterwards, and its deflated Sharpe ratio, 0.72, flags it once the search is counted.

              For VaR 1%, no difference in quantile score from GARCH-t is detected for the ML and pretrained models; GARCH-t has the lowest mean realised QS and QGBM the breach rate closest to 1%. Chronos-2 forecasts without being fitted to Bitcoin and keeps its rank after its release date, but look-ahead in pretrained models must still be ruled out by the design of the test.

              In brief

              • Direction of Bitcoin: ML accuracy 51.3–51.7% against a base rate of 50.6%; Brier scores worse than a constant estimated from past data; XGBoost 78% in sample, 51% out of sample
              • Trading rules: the best of 810 rules in 2018–2022 ranks 779th afterwards; DSR 0.72 once the search is counted
              • VaR: no QS difference from GARCH-t is detected at 1% for the ML and pretrained models; GARCH-t has the lowest mean realised QS, QGBM the breach rate closest to 1%
              • Chronos-2 forecasts without fitting and keeps its rank after its release date; look-ahead must still be ruled out by design

              Interpretation Exercises

              Slide 35 in the PDF

              The five exercises use the numbers of the lecture and follow the format of Part B of the exam: what is measured, what the number says and what it does not say. They cover in-sample against out-of-sample accuracy, the Brier score against a constant, the deflated Sharpe ratio of a selected rule, breach rate against quantile score, and look-ahead in a pretrained model.

              In brief

              1. A random forest classifies tomorrow's Bitcoin direction with 64.6% accuracy in sample and 51.6% out of sample; 50.6% of days are up days. Which number would you report, and against which benchmark?
              2. A model's Brier score is 0.2611 and that of the constant forecast 0.2500. What does this say about the model's probabilities, even if its accuracy exceeds 50%?
              3. A consultant shows a momentum rule with Sharpe ratio 0.84 over five years. You learn it is the best of 810 rules tried. Explain the deflated Sharpe ratio of 0.72 to a client.
              4. QGBM has a 1% breach rate of 1.07%, GARCH-t of 0.68%; the QS is lower for GARCH-t. Which model is better? Why can the two criteria disagree?
              5. A pretrained model released in October 2025 is back-tested on 2024–2025 and beats GARCH. What is your main concern, and how would you address it?
              • Good luck in the exam on 27 November!

              Appendix: Further Material

              The Appendix holds optional material for self-study. It studies a simple correction of a risk forecast with its own past errors: the theory of when it helps, a simulation, and its effect on the six Bitcoin forecasts and on their Basel zones.

              Appendix: Further Material

              Slide 36 in the PDF
              Optional material for self-study.
              • Correcting a Forecast with Its Own Errors
              • Bias against Estimation Cost
              • Monte Carlo Comparison of Raw, Half and Full Recalibration
              • Recalibrating the Bitcoin Forecasts
              • Breach Rate and Loss after Correction
              • Recalibration and the Basel Traffic Light

              Correcting a Forecast with Its Own Errors

              Slide 37 in the PDF

              A biased quantile forecast can be shifted by its own past errors. On a calibration window of \(n\) days the errors are \(e_t = r_t - q_t\); if the forecast were right, \(P(e_t < 0) = \alpha\). The fitted correction \(\hat c = \hat Q_\alpha(e_{t-n}, \dots, e_{t-1})\) is the empirical \(\alpha\)-quantile of the errors, so that \(P(e < \hat c) \approx \alpha\) on the calibration window, and the corrected forecast with intensity \(\lambda\) is

              \[q_t^{(\lambda)} = q_t + \lambda \hat c\]
              (8.12)

              with \(\lambda = 0\) (Raw), \(\tfrac12\) (Half) or 1 (Full). The static version fits \(\hat c\) once on the 500 errors before the test and keeps it fixed; the rolling version refits it every day on the last 500 errors. Full sets the breach rate on the calibration window to about \(\alpha\), exactly only up to ties and interpolation. The method resembles split-conformal calibration, but the coverage guarantee of conformal methods needs exchangeable errors, which a rolling window of a time series does not ensure. The real question is whether the correction lowers the QS on future days; a breach rate closer to \(\alpha\) is not the same thing.

              In brief

              • Forecast errors on a calibration window of \(n\) days: \(e_t = r_t - q_t\). If the forecast were right, \(P(e_t < 0) = \alpha\)
              • Fitted correction: \(\hat c = \hat Q_\alpha(e_{t-n}, \dots, e_{t-1})\), the empirical \(\alpha\)-quantile of the errors; then \(P(e < \hat c) \approx \alpha\) on the calibration window
              • Corrected forecast with intensity \(\lambda\):
                \[ q_t^{(\lambda)} = q_t + \lambda \hat c, \qquad \lambda = 0 \text{ (Raw)}, \;\; \tfrac12 \text{ (Half)}, \;\; 1 \text{ (Full)} \]
              • Static: \(\hat c\) fitted once on the 500 errors before the test and kept fixed. Rolling: refitted every day on the last 500 errors
              • This is a rolling empirical-quantile correction: Full sets the calibration breach rate to about \(\alpha\) (exactly \(\alpha\) only up to ties and interpolation)
              • It resembles split-conformal calibration, but the conformal coverage guarantee needs exchangeable errors, which a rolling window of a time series does not ensure
              • Question: does it lower the QS on future days? A breach rate closer to \(\alpha\) is not the same thing

              Bias against Estimation Cost

              Slide 38 in the PDF

              Let \(\delta\) be the true shift that would minimise the expected QS, the bias of the raw forecast, and \(\hat c = \delta + \varepsilon\) its estimate, with \(V(\varepsilon) = \sigma_Q^2\). With \(f\) the density of the return at the quantile, a second-order expansion gives the excess expected loss \(\Delta(\lambda) \approx \frac f2\left[(1-\lambda)^2\delta^2 + \lambda^2 \sigma_Q^2\right]\): the first term is the bias left uncorrected, the second the noise added by the estimate. Under dependence, \(\sigma_Q^2 \approx \Omega_h/(n f^2)\), where \(\Omega_h\) is the long-run variance of the breach indicator, so the estimation cost falls as \(1/n\). Minimising \(\Delta(\lambda)\) gives the optimal intensity \(\lambda^* = \delta^2/(\delta^2 + \sigma_Q^2)\), a shrinkage factor as in a James–Stein estimator. Comparing the three intensities: Full beats Raw when \(\delta^2 > \sigma_Q^2\), Half beats Full when \(\delta^2 < 3\sigma_Q^2\), Half beats Raw when \(\delta^2 > \sigma_Q^2/3\), and Half is optimal when \(\delta^2 = \sigma_Q^2\). Here \(\delta\) is the bias in the test period: if the bias changes between calibration and test, the fitted \(\hat c\) transfers badly, whatever \(n\) is. Gneiting & Resin (2023) discuss such optimal shifts in the context of calibration diagnostics.

              In brief

              • Let \(\delta\) be the true shift that would minimise the expected QS (the bias of the raw forecast) and \(\hat c = \delta + \varepsilon\), \(\;V(\varepsilon) = \sigma_Q^2\). With \(f\) the density at the quantile, a second-order expansion gives the excess expected loss
                \[ \Delta(\lambda) \approx \frac f2\left[(1-\lambda)^2\delta^2 + \lambda^2 \sigma_Q^2\right] \]
              • Estimation cost under dependence: \(\;\sigma_Q^2 \approx \dfrac{\Omega_h}{n f^2}\), \(\;\Omega_h = \sum_{k=-\infty}^{\infty}\mathrm{Cov}(h_t, h_{t+k})\), the long-run variance of the breach indicator; it falls as \(1/n\)
              • Optimal intensity: \(\;\lambda^* = \dfrac{\delta^2}{\delta^2 + \sigma_Q^2}\) (shrinkage, as in a James–Stein estimator)
              • Comparisons:
                • Full beats Raw \(\iff \delta^2 > \sigma_Q^2\)
                • Half beats Full \(\iff \delta^2 < 3\sigma_Q^2\); Half beats Raw \(\iff \delta^2 > \sigma_Q^2/3\)
                • Half is optimal when \(\delta^2 = \sigma_Q^2\)
              • \(\delta\) refers to the test period. If the bias changes between calibration and test, the fitted \(\hat c\) transfers badly, whatever \(n\) is

              Monte Carlo Comparison of Raw, Half and Full Recalibration

              Slide 39 in the PDF

              The simulation draws independent returns from a Student-\(t\) distribution with 5 degrees of freedom, sets \(\alpha = 1\%\) and \(n = 500\), shifts the raw forecast by a bias \(\delta\) and repeats the experiment 20,000 times. The left panel shows the excess expected QS of Raw, Half and Full against the bias in units of the sampling standard deviation of the correction, \(\delta/\sigma_Q\); the dashed lines give the local theory and the dotted lines its crossing points \(1/\sqrt 3\), 1 and \(\sqrt 3\). The right panel shows the expected breach rate. In the simulation, Half beats Raw from \(\delta/\sigma_Q = 0.65\) (theory 0.58), Half beats Full up to 1.40 (theory 1.73), and Full beats Raw from 0.95 (theory 1). With no bias, Full still raises the expected QS and gives 1.19% breaches instead of 1%, because the 1% empirical quantile of 500 errors rests on 5 observations.

              In brief

              Monte Carlo Comparison of Raw, Half and Full Recalibration
              • Design: returns i.i.d. Student-\(t_5\), \(\alpha = 1\%\), \(n = 500\), raw forecast shifted by \(\delta\), 20 000 replications; dashed lines = local theory
              • Simulated crossings: Half beats Raw from \(\delta/\sigma_Q = 0.65\) (theory 0.58), Half beats Full up to 1.40 (theory 1.73), Full beats Raw from 0.95 (theory 1)
              • With no bias, Full still raises the expected QS and gives 1.19% breaches: the 1% empirical quantile of 500 errors rests on 5 observations
              • Source: own simulation, Student-\(t_5\) returns, 20 000 replications; no market data
              BA_L8_lambda

              Recalibrating the Bitcoin Forecasts

              Slide 40 in the PDF

              The table applies the static correction to the six VaR 1% forecasts. The column \(\hat c^2/\widehat{\mathrm{se}}^2\) compares the squared correction with its squared standard error, from a block bootstrap with blocks of 20 days: by the previous slide, Full can help only when this ratio is clearly above 1. For HS the ratio is 2.18, yet Full raises the breach rate to 3.21% and worsens the QS by 26%, significant at 1% by the DM test against Raw. In mid-2022 – 2023 the 500-day window of HS still held the 2022 crash, so HS was too conservative and \(\hat c > 0\); the correction lowered the VaR just before the more volatile year 2024. At VaR 5% the same correction gives 14.0% breaches. For EWMA-N the Normal quantile is too thin, \(\hat c = -1.14 < 0\) transfers to the test period, and both corrections help, Half significantly at 5%. For GARCH-t, QGBM and Chronos-2 the ratio is near zero, the correction is noise and changes almost nothing. A rolling daily correction limits the damage for HS, with a QS of 0.0908 instead of 0.1058.

              In brief

              \(\alpha = 1\%\)\(\hat c\)\(\hat c^2/\widehat{\mathrm{se}}^2\)breach % (static)QS (static)
              RawHalfFullRawHalfFull
              HS1.912.181.172.043.210.08420.08930.1058\(^{**}\)
              EWMA-N\(-1.14\)0.362.141.660.970.08800.0831\(^{*}\)0.0816
              GARCH-t\(-0.06\)0.000.680.680.680.08030.08040.0805
              QReg0.630.570.880.970.970.08220.08210.0820
              QGBM0.080.011.071.071.170.08480.08480.0849
              Chronos-20.330.051.271.361.360.08270.08330.0839
              • \(\widehat{\mathrm{se}}\): block bootstrap (blocks of 20 days). Stars: DM test against Raw, \(^{*}\) 5%, \(^{**}\) 1%
              • HS: in mid-2022 – 2023 its 500-day window still held the 2022 crash, so it was too conservative and \(\hat c > 0\). The correction moved \(q_t\) up, i.e. lowered the VaR, just before the more volatile 2024: breaches rise to 3.2% and the QS gets 26% worse. At 5% the same correction gives 14.0% breaches
              • EWMA-N: the Normal quantile is too thin; \(\hat c < 0\) transfers, and both corrections help
              • Models with \(\hat c^2/\widehat{\mathrm{se}}^2\) near zero (GARCH-t, QGBM, Chronos-2): the correction is noise and changes almost nothing
              • Rolling daily correction limits the HS damage: QS 0.0908 instead of 0.1058
              • Source: own computation on EODHD daily Bitcoin prices and volumes; test 1 January 2024 – 23 October 2026

              Breach Rate and Loss after Correction

              Slide 41 in the PDF

              The left panel shows the VaR 1% breach rates of each model under Raw, Half and Full, with the 1% target dashed; the right panel shows the change in QS against Raw, in units of \(10^{-3}\), where a positive bar means a worse forecast. The correction moves breach rates in both directions: up from 1.17% to 3.21% for HS, down from 2.14% to 0.97% for EWMA-N. The same tool has opposite outcomes, and the sign depends on whether the bias of the calibration period persists into the test period.

              In brief

              Breach Rate and Loss after Correction
              • Left: correction moves breach rates in both directions. Right: QS change against Raw (\(\times 10^{-3}\)); positive = worse
              • Same tool, opposite outcomes: the sign depends on whether the calibration-period bias persists into the test period
              • Source: own computation on EODHD daily Bitcoin prices and volumes; test 1 January 2024 – 23 October 2026
              BA_L8_recal

              Recalibration and the Basel Traffic Light

              Slide 42 in the PDF

              The Basel backtesting zones for a VaR 1% over 250 days (Basel Committee, 2019) are green for 0–4 breaches, amber for 5–9 and red for 10 or more (amber was called "yellow" in the 1996 rules). The rule behind them puts the start of amber where the binomial probability of the breach count reaches 95% and the start of red where it reaches 99.99%; the same rule for a crypto year of 365 days gives green 0–6, amber 7–12 and red 13 or more. The zones here only illustrate the rule: bank capital is computed for the whole trading book, not from a one-asset VaR. In the table, EWMA-N moves from amber to green in both years under Full, with a VaR 1.1 points higher, that is more conservative. HS moves from green to red in 2024 under Full, with 17 breaches and a lower VaR: a worse model. The zone, the VaR level and the forecast quality measured by the QS are three different outcomes of the same correction, and all three should be reported.

              In brief

              • Basel backtesting zones for a 1% VaR over 250 days (Basel Committee (2019), MAR32): green 0–4, amber 5–9, red 10+ breaches (amber was “yellow” in the 1996 rules)
              • Rule behind them: amber once the binomial probability of the breach count reaches 95%, red at 99.99%. Same rule for a crypto year of 365 days: green 0–6, amber 7–12, red 13+
              • Illustration of the zones only: bank capital is computed for the whole trading book, not from a one-asset VaR
              breaches 2024 (366 days)breaches 2025mean 1% VaR 2024–2025 (%)
              RawHalfFullRawFullRawFull
              HS610173116.34.4
              EWMA-N7631035.76.8
              GARCH-t222337.17.1
              QReg344227.67.0
              QGBM334446.56.4
              Chronos-2788336.76.4
              • Bold = amber (7–12) or red (13+). EWMA-N: Full moves amber to green in both years, with a VaR 1.1 points higher (more conservative)
              • HS: Full moves green to red in 2024 (17 breaches) with a lower VaR: a worse model
              • Zone, VaR level and forecast quality (QS) are three different outcomes of the same correction; report all three
              • Source: own computation on EODHD daily Bitcoin prices and volumes; test 1 January 2024 – 23 October 2026

              References