Modelling Financial Markets

Chapter 13 · Machine Learning in Finance

The slide of the lecture is on the left, the same figure on the right, computed in the browser from the same data. The ▶ button moves the parameter step by step.

What Can Machine Learning Predict in Finance? · Machine Learning Methods · Features: Fractional Differencing and Its Limitations · Labels: Definition of the Target Variable · Validation without Information Leakage · Case Study: Predicting the S&P 500 · Selection and Backtest Overfitting · Machine Learning for Inference · Current Research: Structural and Flexible Models

What Can Machine Learning Predict in Finance?

What Is Predictable? Direction vs. Risk

Slide 6: What Is Predictable? Direction vs. Risk
Slide 6
    Question for the room For the S&P 500 over 2000–2026, daily-return autocorrelations are small at most lags. What do the autocorrelations of absolute returns |r_t| look like?
    Answer

    For |r_t|, autocorrelation is 0.29 at lag 1 and 0.16 at lag 40; all 40 lie outside the ±0.024 band. For returns, the first-order autocorrelation is −0.10, and 13 of 40 lie outside the band. Low linear autocorrelation does not imply independence, and taking absolute values does not simply square the ACF. The result indicates persistence in return magnitudes. The ±0.024 band is a pointwise i.i.d. reference, not a simultaneous band robust to heteroskedasticity.

    Machine Learning Methods

    LASSO Regularisation Path

    Slide 13: LASSO Regularisation Path
    Slide 13
      Question for the room On the LASSO path fitted to the full sample, which feature first becomes non-zero as the penalty decreases?
      Answer

      The first feature is ffd_price. vol_20 has the largest absolute coefficient at a weak penalty, which does not determine entry order. First-order return autocorrelation does not determine it either. The truncated FFD filter retains a component proportional to the log-price level. Early entry does not establish predictive skill: mean purged-CV AUC, with the scaler estimated in each fold, ranges from 0.476 to 0.486 across penalties with at least one non-zero coefficient.

      Features: Fractional Differencing and Its Limitations

      The ADF Statistic Along d: S&P 500

      Slide 19: The ADF Statistic Along d: S&P 500
      Slide 19
        Question for the room At d = 0.25, ADF with one lag rejects the unit root. Is this rejection sufficient to treat the FFD series as stationary?
        Answer

        ADF with one lag rejects: −2.96 < −2.86. With 34 AIC-selected lags, the statistic is −0.72 and does not reject. Neither decision is a proof. The structural argument is that, for an I(1) log price, a finite filter whose weights sum to 0.178 retains the I(1) component. The 0.98 correlation with the log price is a warning sign, not a stationarity test.

        Labels: Definition of the Target Variable

        Triple Barrier: S&P 500 in 2022

        Slide 24: Triple Barrier: S&P 500 in 2022
        Slide 24
          Question for the room An event first hits the profit barrier on day 5. On day 15, the price is below its entry level. Which label does the page’s implementation assign?
          Answer

          The event ends on day 5 and receives +1. Subsequent price movements do not change its label. The return sign at the horizon is used only when neither horizontal barrier has already been hit; conflicting signs are not cancelled. On the chart, only the March event reaches the profit barrier; the May, August and November events receive +1 from the return sign at the vertical barrier, and the January event hits the stop-loss (−1).

          Validation without Information Leakage

          Purged K-Fold with Embargo

          Slide 30: Purged K-Fold with Embargo
          Slide 30
            Question for the room In the scheme with 100 observations and 5 folds, keep the embargo at e = 3. If label length increases from h = 4 to h = 6, how many training observations are excluded around the middle fold?
            Answer

            The number excluded is 2h + e = 15: h = 6 before the block, whose labels end inside the test set, and h + e = 9 after it, i.e. 6 purged and 3 embargoed. The answer 13 updates only the left boundary; 12 omits the embargo. The 20 test observations are not included in this count.

            Leakage Experiment with an Unpredictable Target

            Slide 31: Leakage Experiment with an Unpredictable Target
            Slide 31
              Question for the room In the simulation with independent, symmetric increments and noise features, shuffled K-fold gives 69.4%, versus 49.6% with purging and embargo. What explains the difference?
              Answer

              At h = 20, neighbouring days have overlapping labels and similar features. Shuffling distributes them between training and test. Class imbalance does not explain the difference between evaluations on the same observations, and repeated splitting does not remove this mechanism. Contiguous folds give 50.0%, purging and embargo 49.6%; these values are simulation outcomes, not exact guarantees. With h = 2, shuffled K-fold gives 49.9%; with h = 60 it reaches 84.5%.

              Case Study: Predicting the S&P 500

              Model Comparison: Fold Means

              Slide 38: Model Comparison: Fold Means
              Slide 38
                Question for the room At threshold 0.5, the logit has 57.7% accuracy. Does this result establish an accuracy advantage over the always-"up" rule?
                Answer

                No. The always-"up" rule has 58.1%, so the logit is about 0.4 percentage points below the benchmark. Exceeding 50% or ranking ahead of the other three models does not establish superiority over this benchmark. Predicting "up" on 98.3% of days (TP = 3,575, FP = 2,589) indicates proximity to the majority rule; it does not mean the model reproduces that rule exactly. The fold-mean AUCs lie between 0.477 and 0.508.

                Walk-Forward Backtest: Results

                Slide 42: Walk-Forward Backtest: Results
                Slide 42
                  Question for the room The GBM long/cash strategy has 76% average exposure. Can we treat it as buy-and-hold with constant 76% exposure when comparing Sharpe ratios?
                  Answer

                  No. Sharpe-ratio invariance to scaling concerns multiplying every return by the same positive constant. Switching between positions of 0 and 1 is not such a scaling. In this example, trading at the next close, at threshold 0.5 and a 5 bp cost, the strategy has SR 0.33 and buy-and-hold 0.66. The difference, −0.33, has p = 0.001 (Ledoit–Wolf HAC test). The FFD log price uses d = 0.15, chosen on data up to 2009 only.

                  Anomalies and the Market: Out-of-Sample Gain over the Historical Mean

                  Slide 45: Anomalies and the Market: Out-of-Sample Gain over the Historical Mean
                  Slide 45
                    Question for the room OLS on all 15 anomalies has an R²OOS of −7.41% over 1985–2026. Why does the mean of the 15 univariate forecasts do better, with 0.41%?
                    Answer

                    Comb averages univariate forecasts: each univariate slope contributes with weight 1/15. It is not multivariate OLS divided by 15. Combining forecasts can reduce error variance without guaranteeing a gain. Multicollinearity alone does not bias OLS. In this sample, Comb has an R²OOS of 0.41%, versus −7.41% for OLS; the significance of Comb’s gain is sensitive to the HAC lag choice (one-sided Clark–West p 0.045 with 5 lags, 0.063 with 3 lags). In the paper, with 100 anomalies, OLS reaches −2,513.86% and the simple combination 0.89%.

                    Selection and Backtest Overfitting

                    The Best of N Zero-Skill Strategies

                    Slide 49: The Best of N Zero-Skill Strategies
                    Slide 49
                      Question for the room For 1,000 independent zero-skill strategies, each evaluated on 5 years of i.i.d. daily returns, what is the approximate expected maximum annualised Sharpe ratio?
                      Answer

                      About 1.45: the theoretical approximation gives 1.46, and the mean of 200 simulations is 1.43. Zero is the mean for an individual strategy; 0.45 = √(1/5) is its approximate standard deviation. The realised maximum varies across samples. The value rises with N and falls with the sample length.

                      In-Sample (IS) vs. Out-of-Sample (OOS): Bitcoin Grid Search

                      Slide 51: In-Sample (IS) vs. Out-of-Sample (OOS): Bitcoin Grid Search
                      Slide 51
                        Question for the room Using only May 2015–December 2020, we select the rule with the highest in-sample SR, 2.02. What SR does this rule achieve in 2021–2026?
                        Answer

                        The MA 11/20 rule with a 0% band has an out-of-sample SR of 0.03. The values 0.50 and 0.41 describe group medians (the top 10% in-sample rules and all rules), not this rule’s performance. The Spearman rank correlation between in-sample and out-of-sample SR is 0.13. The decline is consistent with overfitting; comparison with buy-and-hold requires a separate evaluation.

                        Machine Learning for Inference

                        Worked Example: Does the VIX Predict S&P 500 Returns?

                        Slide 59: Worked Example: Does the VIX Predict S&P 500 Returns?
                        Slide 59
                          Question for the room OLS on log VIX alone gives 7.5 bp, with interval [−10.3, 25.2]. Why is the double-selection interval almost twice as wide?
                          Answer

                          With 4 HAC lags, the double-selection interval, [−22.1, 40.8] bp, is about 1.8 times wider. The 15 controls selected for log VIX reduce its remaining variation after adjustment, which can increase the standard error. The methods use the same sample, and both intervals use HAC correction. Coefficient magnitude does not proportionally determine interval width. The [−22.1, 40.8] bp interval does not distinguish the predictive coefficient from zero at the 5% level.

                          Current Research: Structural and Flexible Models

                          Deep Learning in Asset Pricing: Out-of-Sample Sharpe Ratios

                          Slide 63: Deep Learning in Asset Pricing: Out-of-Sample Sharpe Ratios
                          Slide 63
                            Question for the room With FF5 weights estimated on 1967–1986 and then held fixed, what is the monthly Sharpe ratio over the 1992–2016 test period?
                            Answer

                            FF5 has a monthly SR of 0.22, with an approximate 95% interval of [0.11, 0.34]. The page uses standard error √((1 + SR²/2)/T), under an i.i.d. Normal approximation. The values 0.75 and 0.15 belong to the paper’s GAN and the replication’s market benchmark, respectively. In the paper, EN achieves 0.50 and FFN 0.44: the regularised linear model outperforms this forecasting network in the reported comparison, without establishing a universal rule about structure and flexibility.

                            S&P 500 and VIX: daily closes, January 2000 – September 2026; Bitcoin: daily closes from September 2014; EODHD prices. Fama–French factors: Kenneth French Data Library. Sharpe ratios of the SDF models: Chen, Pelger and Zhu (2024), Table I and Table A.VI. Long-short anomalies (factors and decile spreads): Kenneth French Data Library; R²OOS of the paper: Dong, Li, Rapach and Zhou (2022), Table 2.