Neural Networks and Deep Learning

Lecture 10 · Reinforcement learning: trading, hedging and battery dispatch

Reinforcement learning (RL) on three real problems: a 2 MWh (megawatt-hour) battery trading on the day-ahead power market, a position in the S&P 500 index that is either flat or long, and the daily hedge of a one-month call option. The slide of the lecture is on the left, the same figure on the right, recomputed in your browser from the course data (data/clean) and from the decisions of the trained agents when you move a control. The ▶ button moves the main parameter step by step. Under each figure: a readout, a question for the room and the Quantlet with the code of the chart.

The battery as a Markov decision process (MDP): dynamic programming (DP) · Value-based learning: Q-learning and the deep Q-network (DQN) · Policy gradients: proximal policy optimisation (PPO) for trading · Deep hedging of a short call

The battery as a Markov decision process (MDP): dynamic programming (DP)

Optimal battery schedule for one day

Slide 16: Optimal battery schedule for one day
Slide 16

    Code of this chart: NN_L10_dp_day · open in Colab

    Question for the room On 30 June 2026 the Romanian day-ahead price ranged from 100.65 EUR/MWh at 13:00 to 954.59 EUR/MWh at 20:00, and a 2 MWh battery earned 1,541 EUR with perfect foresight. Does a 4 MWh battery earn twice as much?
    Answer

    Almost, not quite: 2,885 EUR, 87% more. The battery moves 1 MWh per hour, so the third and fourth MWh are bought in less cheap hours and sold in less dear ones; with 6 MWh the revenue is 3,282 EUR, only 14% above 4 MWh, because the evening peak lasts a few hours. A perfect battery (100% round trip) would earn 1,745 EUR at 2 MWh. On the same day Hungary gives 1,466 EUR, Germany–Luxembourg 708 EUR and Bulgaria 744 EUR: the revenue follows the daily spread, not the price level. All these numbers are upper bounds, because dynamic programming (DP) sees the 24 prices of the day before they are known.

    The value function prices stored energy hour by hour

    Slide 17: The value function prices stored energy hour by hour
    Slide 17

      Code of this chart: NN_L10_dp_value · open in Colab

      Question for the room Go back hour by hour with ▶. On 30 June 2026, what is one stored MWh worth at 12:00, at 18:00 and at 23:00?
      Answer

      At 12:00, V12(0) = 1,541 EUR and V12(2) = 1,760 EUR, so each stored MWh is worth 109.38 EUR: the shadow price of energy in the battery, about what it costs to buy it at midday. At 18:00 each MWh is worth about 880 EUR, because the two dearest hours (19:00 and 20:00) are still ahead. At 23:00 (switch to the value of one more stored MWh) the first MWh is worth 211 EUR (it can still be sold in the last hour) and the second nothing, since only 1 MWh can be moved per hour. The Bellman equation builds these values backwards from V24(s) = 0, and the optimal action at each (h, s) is the one that attains the maximum.

      Value-based learning: Q-learning and the deep Q-network (DQN)

      The policy learned by tabular Q-learning

      Slide 22: The policy learned by tabular Q-learning
      Slide 22
      Stored energy

        Code of this chart: NN_L10_q_policy · open in Colab

        Question for the room The tabular agent does not see today's prices, only how each hour ranked yesterday. At 13:00, with 1 MWh stored, in an hour that was among yesterday's six cheapest, which action does it choose, and how sure is it?
        Answer

        It charges: Q(charge) = 175.0 EUR, Q(idle) = 167.2 EUR, Q(discharge) = 139.6 EUR, the values of the rest of the day after each action, learned from 1,746 training days (2019–2023). The margin over idling is small, about 8 EUR. With 1 MWh stored, the row of yesterday's cheapest hours charges in 7 hours and discharges in 8, mostly from 18:00 on: the hour of the day dominates, as in the lecture. With 0 MWh the agent can only charge or wait, with 2 MWh only discharge or wait (action masking). The table has 24 × 4 × 3 = 288 states and can be printed and audited.

        Share of the perfect-foresight revenue captured in the test years

        Slide 25: Share of the perfect-foresight revenue captured in the test years
        Slide 25
        Strategies

          Code of this chart: NN_L10_battery_test_cum · open in Colab

          Question for the room On the 638 test days, 1 January 2025 to 24 October 2026, which captures more of the perfect-foresight revenue: the double deep Q-network (DQN), tabular Q-learning, or simply repeating yesterday's optimal plan?
          Answer

          Yesterday's plan: 163.1 thousand EUR, 80.7% of the 202.1 thousand EUR upper bound. Tabular Q-learning captures 115.2 thousand EUR (57.0%) and the double DQN 101.3 thousand EUR (50.1%, mean of 5 seeds; choose the seeds one by one to see the spread between them). The DQN beats the naive plan on only 9% of the days. The learning agents see a coarse summary of yesterday (price quartiles and a few features), the naive plan the full 24-hour profile of yesterday, and prices are strongly autocorrelated from one day to the next: a simple benchmark is hard to beat.

          Policy gradients: proximal policy optimisation (PPO) for trading

          Test period: the PPO agent against two simple rules

          Slide 33: Test period: the PPO agent against two simple rules
          Slide 33
          Strategies

            Code of this chart: NN_L10_trading_equity · open in Colab

            Question for the room With a cost of 5 basis points (bp) per trade, the proximal policy optimisation (PPO) agent has a higher Sharpe ratio than buy and hold over 2020–2026 (0.83 against 0.61, mean of the five seeds). Does the advantage survive higher costs or a later start?
            Answer

            Only partly. At 20 bp the mean Sharpe ratio of the seeds falls to 0.63, level with buy and hold (0.61); at 30 bp it is 0.50, and two seeds that trade often drop to 0.02 and 0.07. Starting in 2023, buy and hold has a Sharpe ratio of 1.23 against 1.04 for PPO at 5 bp: the advantage comes from 2020 and 2022, when the agent was partly out of the market (exposure chart). The seeds differ (0.72 to 0.95 at 5 bp) and the block-bootstrap interval of the Sharpe difference in the lecture, −0.08 to 0.56, contains zero. One backtest of an RL agent is not evidence of skill.

            Deep hedging of a short call

            Hedging profit and loss on the test paths

            Slide 40: Hedging profit and loss on the test paths
            Slide 40
            Strategies

              Code of this chart: NN_L10_hedge_pnl · open in Colab

              Question for the room A desk sells a one-month at-the-money call on the S&P 500 and hedges it daily with 10 bp costs. By how much does the deep hedge cut the expected shortfall (ES) of the loss compared with the Black-Scholes delta, and is the gain the same in calm and in stressed markets?
              Answer

              Over the 1,696 test paths (start days 2020–2026), the 95% ES of the loss is 6.82% of the index level without a hedge, 1.94% with the Black-Scholes delta and 1.15% with the deep hedge (mean of 5 seeds). The standard deviations are close (0.74% and 0.69%): the network was trained to shrink the tail, not the variance. The gain is larger in stress: with VIX below 20 at the start (986 paths) the ES falls from 1.41% to 1.01%, with VIX of 20 or more (710 paths) from 2.60% to 1.31%. At 99% the ES is 4.09% against 2.43%.

              Hourly day-ahead prices: OPCOM (Romania), HUPX (Hungary), EPEX (Germany–Luxembourg), IBEX (Bulgaria), via energy-charts.info (Fraunhofer ISE); delivery hours in Central European Time (CET/CEST), days with 24 complete hours; since October 2025 each hour is the mean of four 15-minute prices. S&P 500 and Cboe Volatility Index (VIX): Yahoo Finance, daily closes. The agents were trained once by lectures/L10/analysis_L10.py (five seeds, 20261108–20261112); the page replays their test-period decisions and recomputes the dynamic programme, the costs and the risk measures. Data to the cut-off of the lecture, 31 October 2026.