Neural Networks and Deep Learning

Lecture 12 · Large language models (LLM) in finance and prompt engineering

The slide of the lecture is on the left, the same figure on the right, recomputed in your browser from the outputs of the open models run for the lecture (Qwen2.5 with 0.5, 1.5 and 3 billion parameters) when you move a control. Nothing is invented: every probability, label and Value-at-Risk (VaR) estimate below was produced by the models on real data. The ▶ button moves the main parameter step by step. Under each figure: a readout, a question for the room and the Quantlet with the code of the chart.

Next-token prediction in a language model · Evaluation of prompt engineering · Retrieval-augmented generation (RAG) · A language model as a risk estimator (LLM-VaR)

Next-token prediction in a language model

Next-token probabilities of a real model on a real sentence

Slide 10: Next-token probabilities of a real model on a real sentence
Slide 10

    Code of this chart: NN_L12_next_token · open in Colab

    Question for the room The sentence of an ECB (European Central Bank) speech of 30 October 2026 starts “The inflation impact is limited:”. At temperature 1, the model gives 10.1% to “the”. What happens to this probability at temperature 0.5 and at temperature 2?
    Answer

    Dividing the logits by T sharpens the distribution when T < 1 and flattens it when T > 1. Over the whole vocabulary of 151,665 tokens the lecture measures 49.9% for “the” at T = 0.5 and only 0.4% at T = 2; the entropy rises from 1.74 to 6.20 and 10.29 nats. The chart renormalises over the k most likely tokens, as top-k sampling does: with k = 1 the model always writes “the” (greedy decoding). The word actually spoken, “prices”, is not among the eight most likely tokens. For classification and extraction use T = 0; high temperatures are for brainstorming.

    Evaluation of prompt engineering

    Effect of the number of examples in the prompt

    Slide 24: Effect of the number of examples in the prompt
    Slide 24
    Set of examples (random draw)

      Code of this chart: NN_L12_fewshot_curve · open in Colab

      Question for the room Adding labelled examples to the prompt (few-shot) is supposed to help. Does the 3-billion-parameter model classify the Financial PhraseBank sentences better with 4 examples than with none?
      Answer

      No. Zero-shot, Qwen2.5-3B reaches a macro-F1 of 92.2%; with k = 1, 2, 4 and 8 examples the mean over three random draws of examples is 89.5%, 90.4%, 92.1% and 91.3%, and with k = 1 the draws range from 87.5% to 91.1%. The smaller 1.5B model gains a little at k = 1 (85.1% → 87.3%) and loses at k = 8 (84.1%). Which examples you draw matters about as much as how many. Macro-F1 averages the F1 score (harmonic mean of precision and recall) of the three classes, so the rare negative class counts as much as the neutral one.

      Prompt sensitivity: effect of rewording the instruction

      Slide 25: Prompt sensitivity: effect of rewording the instruction
      Slide 25
      Wording of the instruction

        Code of this chart: NN_L12_prompt_sensitivity · open in Colab

        Question for the room Five paraphrases of the same instruction, same model, same sentences. How much can the wording alone move the macro-F1 score?
        Answer

        A lot for the smaller model: on Financial PhraseBank, Qwen2.5-1.5B ranges from 72.8% to 87.0% (14.2 points) across the five wordings, Qwen2.5-3B only from 88.5% to 92.2% (3.6 points). On finance tweets both are fragile: 34.3–47.6% for 1.5B and 65.5–78.4% for 3B. The confusion matrix shows where the errors go; a reported LLM score is a score for one prompt, so test several wordings and report the range.

        Calibration of the label probabilities on real data

        Slide 28: Calibration of the label probabilities on real data
        Slide 28

          Code of this chart: NN_L12_calibration · open in Colab

          Question for the room A classifier that abstains when unsure can hand the hard cases to a human analyst. If the zero-shot 3B model answers only when its probability is high, does its accuracy on the answered sentences rise?
          Answer

          Yes: move the threshold and watch coverage fall while accuracy on the answered sentences rises. This works only if the probabilities are calibrated, that is, if “90% sure” is right about 90% of the time. The lecture measures the expected calibration error (ECE) on Financial PhraseBank: 4.7% for zero-shot 3B, 3.6% for few-shot 3B and 5.4% for TF-IDF (term frequency–inverse document frequency) with logistic regression. Compare the 0.5B model: its reliability curve is far from the diagonal, and a threshold buys much less.

          Retrieval-augmented generation (RAG)

          Retrieval of the correct speech from its title

          Slide 36: Retrieval of the correct speech from its title
          Slide 36
          Retriever

            Code of this chart: NN_L12_rag_recall · open in Colab

            Question for the room The query is the title of an ECB speech; the retriever must find that speech among 400 (8,400 chunks of 200 words). Which retriever puts the right speech first most often?
            Answer

            The hybrid one. Recall@1 is 35.2% for TF-IDF, 37.0% for BM25 (Best Matching 25, a keyword score), 44.0% for dense embeddings and 50.7% for the hybrid that fuses BM25 and dense ranks; at k = 10 the four reach 75.0%, 78.0%, 80.0% and 83.2%. Keywords and meaning fail on different queries, so combining them helps. With retrieval, the 3B model answers “who gave this speech?” correctly for 81.2% of 160 recent speeches, against 13.8% from its own memory.

            A language model as a risk estimator (LLM-VaR)

            LLM-VaR with a 3B open model on the S&P 500

            Slide 41: LLM-VaR with a 3B open model on the S&P 500
            Slide 41
            VaR level (tail probability)
            Methods

              Code of this chart: NN_L12_llm_var · open in Colab

              Question for the room A language model reads the last 250 daily returns of the S&P 500 as text and writes a number: tomorrow’s 5% Value-at-Risk. Over 459 trading days we expect about 23 violations. How does it compare with GARCH and historical simulation?
              Answer

              At 5% the LLM-VaR has 28 violations (6.1%, Kupiec p = 0.295), historical simulation 23 (5.01%, p = 0.991) and GARCH(1,1) with Student-t errors 35 (7.63%, p = 0.016, rejected at 5%). At 1% the counts are 4, 6 and 9 for an expected 4.6. The Kupiec test asks whether the share of violations equals the VaR level; a p-value below 0.05 rejects the model. The LLM passes the count test, but its VaR barely reacts to volatility (correlation with GARCH 0.05): move the start of the period and see where its violations cluster.

              Next token and RAG: speeches of the European Central Bank (ECB), 2022–2026. Sentiment: Financial PhraseBank (Malo et al., 2014; sentences with 100% annotator agreement, test set of 678) and Twitter financial news (Zeroshot, test sample of 900 tweets). Models: Qwen2.5-Instruct (Alibaba), run locally; label probabilities = probability mass of the first token of each label. LLM-VaR: daily log returns of the S&P 500 index, 2 January 2025 to 30 October 2026, window of 250 days, after Pele et al. (2026).