Neural Networks and Deep Learning

Lecture 12: Large language models in finance and prompt engineering

Lecture 12

Large language models in finance and prompt engineering

Prof. dr. Daniel Traian PELE

Bucharest University of Economic Studies (ASE), Department of Statistics and Econometrics
Romanian Academy, Institute for Economic Forecasting (IPE)

Text is the largest data set of a financial institution that is still read mostly by people: news, central-bank speeches, annual reports, loan files. Large language models (LLM) read text at scale, but they are also confident, expensive and easy to fool. This lecture treats an LLM as what it is, a next-token probability model, and measures every claim on labelled financial data with open models that run on a laptop.

We start from the language-model objective, tokens, perplexity and temperature, measured on 400 speeches of the European Central Bank (ECB). We then explain pre-training, instruction tuning and alignment. The core of the lecture is prompt engineering as a measurable activity: zero-shot, few-shot and chain-of-thought prompts on the Financial PhraseBank and on finance tweets, with bootstrap intervals, seeds, paraphrases, calibration and cost. We build retrieval-augmented generation (RAG) over the ECB speeches and measure hallucination, and we backtest a language model used for the Value-at-Risk (VaR), after the LLM-VaR framework of our group.

What you will be able to do

  • Write the language-model objective, perplexity and the temperature of sampling, and measure them on real text.
  • Explain pre-training, instruction tuning and alignment from human preferences, with their loss functions.
  • Turn a prompt into a classifier and measure zero-shot, few-shot and chain-of-thought prompts with confidence intervals.
  • Build and evaluate retrieval-augmented generation over real speeches, and measure hallucination.
  • Backtest a language model used for the Value-at-Risk and know the risks: leakage, cost, calibration.

Lecture plan

  1. Next-token prediction and large language modelsAn LLM is trained to predict the next token.
  2. Pre-training, instruction tuning and alignment
  3. Evaluation of prompt engineering on labelled dataA prompt is a model.
  4. Retrieval-augmented generation on ECB speeches
  5. Language models as risk models: LLM-VaR
  6. Risks and governance

Next-token prediction and large language models

An LLM is trained to predict the next token. Everything else, from answering questions to writing a risk report, is repeated use of that single probability.

Text data in financial institutions

Slide 5 in the PDF

Prices react to news within minutes, central banks steer expectations with words, and loan files and annual reports cost analyst hours. All models in this lecture are open and run locally (Qwen2.5-Instruct with 0.5, 1.5 and 3 billion parameters), so no text leaves the machine and no application programming interface (API) is paid.

The language-model objective

Slide 6 in the PDF
\[p_{\boldsymbol\theta}(x_1,\dots,x_T)=\prod_{t=1}^{T}p_{\boldsymbol\theta}(x_t\mid x_{<t}),\qquad \mathcal L=-\frac{1}{T}\sum_{t=1}^{T}\log p_{\boldsymbol\theta}(x_t\mid x_{<t}),\qquad \mathrm{PPL}=e^{\mathcal L}\]

Perplexity is the effective number of equally likely next tokens. On ECB speeches published after July 2025 (first 1,024 tokens of 40 speeches) the perplexity is 14.71 for the 0.5B model, 10.63 for 1.5B and 9.43 for 3B.

Subword tokens

Slide 7 in the PDF

Byte-pair encoding (BPE) starts from bytes and repeatedly merges the most frequent adjacent pair until the vocabulary is full: Qwen2.5 has 151,665 tokens; the WordPiece vocabulary of FinBERT, a financial version of bidirectional encoder representations from transformers (BERT), has 30,522. On the Financial PhraseBank there are 1.35 Qwen tokens per word.

Sentence length in words and in tokens

Slide 8 in the PDF
Chart: Sentence length in words and in tokens: two tokenisers

Length of the Financial PhraseBank sentences in words and in tokens.

Source: Financial PhraseBank (Malo et al., 2014). Data: 4,838 sentences.

A sentence has on average 23.1 words and 31.1 Qwen tokens; an ECB speech has on average 4,143 tokens. Costs and context limits are counted in tokens.

\[p_T(v\mid x_{<t})=\frac{\exp(z_v/T)}{\sum_u\exp(z_u/T)},\qquad H_T=-\sum_v p_T(v)\log p_T(v)\]

With the prompt "The inflation impact is limited:", taken from the ECB speech of 30 October 2026, the most likely next token "the" has probability 49.9 % at temperature 0.5, 10.1 % at 1 and 0.4 % at 2; the entropy grows from 1.74 to 10.29 nats.

Next-token probabilities of a real model

Slide 10 in the PDF
Chart: Next-token probabilities of a real model on a real sentence

Top next tokens of Qwen2.5-1.5B-Instruct at three temperatures.

Source: European Central Bank, speeches dataset. Data: prompt from the ECB speech of 30 Oct 2026; model Qwen2.5-1.5B-Instruct.

In practice: for classification and extraction use temperature 0; reserve high temperatures for generating ideas.

The loss falls as a power of model size and data, \(\mathcal L(P,D)=E+A/P^{\alpha}+B/D^{\beta}\) (Kaplan et al., 2020; Hoffmann et al., 2022). On our three model sizes the slope of log loss on log parameters is −0.100: three points describe our sample, they are not a law.

Training-data cut-off

Slide 12 in the PDF
Chart: Training-data cut-off and the apparent ease of older speeches

Perplexity on ECB speeches of 2022–2023 and from July 2025.

Source: European Central Bank, speeches dataset. Data: 40 + 40 speeches, first 1,024 tokens of each.

For the 3B model, perplexity is 8.90 on speeches of 2022–2023 and 9.43 on speeches from July 2025, after the model's training data. Topics also changed, so the gap is a symptom, not a proof, of memorisation; it is a reason to evaluate LLMs on documents written after their training cut-off.

Pre-training, instruction tuning and alignment

Three training stages

Slide 14 in the PDF

A generative pre-trained transformer (GPT) is first trained on trillions of tokens with the next-token loss; supervised fine-tuning (SFT) then teaches the format of answers, and reinforcement learning from human feedback (RLHF) or direct preference optimisation (DPO) teaches which answers people prefer (Ouyang et al., 2022).

Instruction tuning

Slide 15 in the PDF
\[\mathcal L_{\text{SFT}}(\boldsymbol\theta)=-\sum_{(\mathbf x,\mathbf y)}\sum_{t=1}^{|\mathbf y|}\log p_{\boldsymbol\theta}(y_t\mid\mathbf x,\mathbf y_{<t})\]

The loss is computed on the answer tokens only. Low-rank adaptation (LoRA) trains \(\mathbf W=\mathbf W_0+\mathbf B\mathbf A\) with a small rank \(r\): a 4096 × 4096 layer with \(r=8\) has 65,536 trainable weights instead of 16.8 million (Hu et al., 2022).

People choose the better of two answers; a reward model \(r_\phi\) is fitted with \(\Pr(y_w\succ y_l)=\sigma(r_\phi(x,y_w)-r_\phi(x,y_l))\) and the policy maximises the reward with a Kullback–Leibler penalty that keeps it close to the SFT model. DPO optimises the same objective directly from the preference pairs (Rafailov et al., 2023). Alignment sets tone and safety; it does not add financial knowledge.

The open models used here

Slide 17 in the PDF

Qwen2.5-Instruct with 0.49, 1.54 and 3.09 billion parameters (Qwen Team, 2024), the bge-small embedding model for retrieval, and a term frequency–inverse document frequency (TF-IDF) representation with logistic regression as the classical benchmark.

Evaluation of prompt engineering on labelled data

A prompt is a model. It is chosen on validation data and scored once on a held-out test set, with the same statistics as any classifier.

A prompt as a classifier

Slide 20 in the PDF
\[\hat p(c\mid x)=\frac{\sum_{v\in\mathcal V_c}p_{\boldsymbol\theta}(v\mid\mathrm{prompt}(x))}{\sum_{c'}\sum_{v\in\mathcal V_{c'}}p_{\boldsymbol\theta}(v\mid\mathrm{prompt}(x))},\qquad \hat c=\arg\max_c\hat p(c\mid x)\]

where \(\mathcal V_c\) are the first tokens of the label word \(c\) in all its spellings. One forward pass per sentence is enough, and the shared instruction and examples are encoded once.

Two labelled data sets

Slide 21 in the PDF

The Financial PhraseBank (Malo et al., 2014) contains 4,838 sentences from news about Finnish listed companies; we use the 2,259 with unanimous labels, split into 1,581 training and 678 test sentences. The Twitter financial news data set gives 9,543 training tweets and a stratified test sample of 900 tweets.

Prompted LLMs against a trained benchmark

Slide 22 in the PDF
Chart: Prompted LLMs against a trained benchmark

Macro-F1 with 95 % bootstrap intervals on both data sets.

Source: Financial PhraseBank (Malo et al., 2014) and Twitter financial news (zeroshot). Data: test sets: 678 sentences (100 % agreement) and 900 tweets; 95 % bootstrap intervals.

MethodPhraseBank macro-F1 (%)Twitter macro-F1 (%)
TF-IDF + logistic regression86.975.5
Zero-shot 0.5B31.018.7
Zero-shot 1.5B85.140.9
Zero-shot 3B92.277.4
Few-shot 3B (12 examples)92.179.8
Chain-of-thought 3B88.465.1
LLM embeddings + logistic87.579.0

On PhraseBank the 3B model reaches 92.2 % macro-F1 zero-shot, above the TF-IDF classifier trained on 1,581 labelled sentences (86.9 %). On tweets, which are shorter and noisier, the best results come from the 3B model with examples (79.8 %) and from LLM embeddings with a trained classifier (79.0 %). The 0.5B model is close to the majority class.

Accuracy and macro-F1 with bootstrap intervals

Slide 23 in the PDF

Macro-F1 averages the three per-class F1 scores, so the small negative class counts as much as the neutral one. The 95 % intervals come from 1,000 bootstrap resamples of the test set: for zero-shot 3B on PhraseBank the interval is [89.8; 94.4] %, for TF-IDF [83.8; 90.0] %.

Examples in the prompt

Slide 24 in the PDF
Chart: Effect of the number of examples in the prompt

Macro-F1 against the number of examples in the prompt, three seeds.

Source: Financial PhraseBank (Malo et al., 2014). Data: test set of 678 sentences; bars = range over 3 seeds.

In-context learning changes the output without changing any weight. For the 1.5B model, macro-F1 moves from 85.1 % with no example to 84.1 % with 24; for the 3B model from 92.2 % to 91.3 %. On this task the zero-shot instruction is already strong, and the choice of examples moves the score by several points.

Prompt sensitivity

Slide 25 in the PDF
Chart: Prompt sensitivity: effect of rewording the instruction

Macro-F1 under five paraphrases of the same instruction.

Source: Financial PhraseBank and Twitter financial news. Data: test sets as above; five paraphrases of the instruction.

Five instructions with the same meaning give a range of 3.6 points on PhraseBank and 12.9 points on Twitter for the 3B model, and 14.2 points on PhraseBank for the 1.5B model. The wording of a prompt is a hyper-parameter: select it on validation data and freeze it.

Chain-of-thought

Slide 26 in the PDF

Letting the model write two sentences of reasoning before the label (Wei et al., 2022) gives 88.4 % macro-F1 on PhraseBank and 65.1 % on Twitter, below the direct zero-shot answer, at about 389 ms per sentence instead of 48.5 ms. Chain-of-thought helps multi-step reasoning, not one-word labels on short sentences.

Chart: Calibration of the label probabilities on real data

Reliability diagram of the predicted label probabilities.

Source: Financial PhraseBank (Malo et al., 2014). Data: test set of 678 sentences, 10 bins.

The expected calibration error (ECE) is 4.7 % for zero-shot 3B, 3.6 % for few-shot 3B and 5.4 % for TF-IDF with logistic regression: on this task the label probabilities of the LLM are about as reliable as those of the trained classifier. Calibration must still be checked on validation data before scores are used as risk probabilities (Guo et al., 2017).

Accuracy by level of annotator agreement

Slide 29 in the PDF
Chart: Accuracy by level of annotator agreement

Accuracy by annotator agreement.

Source: Financial PhraseBank (Malo et al., 2014). Data: sentences never used for training or for examples.

Zero-shot 3B is right on 93.1 % of the unanimous test sentences but only on 53.7 % of the sentences on which half of the annotators disagreed. A model cannot be more certain than the people who labelled the data.

The confusion matrix

Slide 30 in the PDF

For the best prompted model on PhraseBank, 20 neutral sentences were read as positive and 13 positive sentences as neutral, while positive and negative were rarely swapped. Errors sit on the neutral border.

Cost against accuracy

Slide 31 in the PDF
Chart: Cost against accuracy: milliseconds per sentence on a laptop

Macro-F1 against milliseconds per sentence on a laptop.

Source: Financial PhraseBank (Malo et al., 2014). Data: test set of 678 sentences; time measured on a laptop graphics processor.

TF-IDF needs 2.50 ms per sentence, zero-shot 3B 48.5 ms and chain-of-thought about 389 ms. With labels, a small trained model is hard to beat; without labels, a prompt is a strong start; for millions of documents, cost decides.

LLM embeddings with a small classifier

Slide 32 in the PDF

Averaging the last hidden states of Qwen2.5-1.5B and fitting a logistic regression on top gives 87.5 % on PhraseBank and 79.0 % on Twitter: the cheapest form of fine-tuning.

Retrieval-augmented generation on ECB speeches

Retrieval followed by generation

Slide 34 in the PDF
\[p(y\mid q)\approx\sum_{z\in\mathrm{top}_k(q)}p_\eta(z\mid q)\,p_{\boldsymbol\theta}(y\mid q,z)\]

The corpus has 400 ECB speeches cut into 8,400 chunks of 200 words (Lewis et al., 2020).

Three retrievers

Slide 35 in the PDF

The Best Matching 25 (BM25) score weights shared words by their rarity and saturates repeated counts; dense retrieval compares embeddings by cosine similarity; the hybrid combines both rankings by reciprocal rank fusion. Mean reciprocal rank (MRR): BM25 0.515, dense 0.567, hybrid 0.622.

Retrieval of the correct speech from its title

Slide 36 in the PDF
Chart: Retrieval of the correct speech from its title

Recall@k of the four retrievers.

Source: European Central Bank, speeches dataset. Data: 400 speeches, 8400 chunks; query = speech title.

The hybrid retriever puts the right speech first for 50.7 % of the titles and within the first ten for 83.2 %.

Answers with and without retrieved documents

Slide 37 in the PDF
Chart: Answers with and without retrieved documents

Who gave the speech? Answers without documents, with RAG and with the right speech.

Source: European Central Bank, speeches dataset. Data: 160 single-speaker speeches from 15 Jan 2025 to 30 Oct 2026; model Qwen2.5-3B-Instruct.

On 160 speeches given after the model's training data, a forced answer without documents is wrong in 86.2 % of cases (the model answered "Philip Lane" 137 times). When "I don't know" is allowed, it abstains on 100.0 % of questions. With the top-3 retrieved speeches it answers correctly in 81.2 % of cases, and with the right speech in 98.8 %.

In practice: a confident wrong name is the costly failure: require sources in every answer and check them.

Designing a RAG system for a bank

Slide 38 in the PDF

Chunk size, metadata that travel with each chunk, a hybrid retriever (recall@3 of 73.1 % on the question set), permission to abstain and a fixed evaluation set rerun after every change are the choices that matter.

Language models as risk models: LLM-VaR

In the LLM-VaR framework (Pele et al., 2026) a general-purpose model reads the last 250 daily returns in a prompt and writes the Value-at-Risk, with no estimation and no fine-tuning. We test the idea with an open 3B model on the S&P 500, from 2 January 2025 to 30 October 2026 (459 days), against historical simulation (HS) and a GARCH(1,1) model with Student-t errors, with the Kupiec test of coverage.

LLM-VaR with a 3B open model

Slide 41 in the PDF
Chart: LLM-VaR with a 3B open model on the S&amp;P 500

S&P 500 returns with the 5 % LLM-VaR and the GARCH VaR.

Source: Yahoo Finance, daily quotes. Data: 2 Jan 2025 to 30 Oct 2026, S&P 500, trading days.

At 5 %, the LLM-VaR was breached 28 times, GARCH 35 times and historical simulation 23 times, against 23.0 expected. The mean 5 % VaR of the small model is −1.99 % against −1.42 % for GARCH, and the two series are almost uncorrelated (0.05): the 3B model reaches its coverage by being cautious, not by tracking volatility.

ModelViolations 1 %Kupiec p (1 %)Violations 5 %Kupiec p (5 %)
LLM-VaR (Qwen2.5-3B)40.777280.295
Historical simulation60.528230.991
GARCH(1,1)-t90.067350.016

The published study evaluated much larger general-purpose models; the results here describe a 3B model on a laptop only.

Risks and governance

Six risks and the test for each

Slide 44 in the PDF

Hallucination (86.2 % wrong names without documents) is controlled by RAG with sources; leakage by testing on text written after the training cut-off; prompt sensitivity (12.9 points) by selecting and freezing prompts on validation data; miscalibration (ECE 4.7 % zero-shot) by checking and, if needed, recalibrating on validation data; cost by the smallest model that passes the test; data protection by local models and by the duties of the European Union (EU) Artificial Intelligence (AI) Act for high-risk uses such as credit scoring.

Explore it visually

Free interactive tools and illustrated guides for this lecture; most run in the browser.

  • Transformer Explainer: the temperature of this lecture on a live GPT-2: at low temperature one continuation takes almost all the probability, at high temperature unlikely tokens become plausible.
  • Mixture of experts (M. Grootendorst): a router sends each token to a few experts, so a model can have many parameters while using only some of them per token.
  • LLM Inference Handbook: what happens between the prompt and the answer: the prefill of the prompt, token-by-token decoding, the cache of keys and values, and the time to the first token.
  • LLM Visualization (B. Bycroft): the same architecture at several sizes, from a small GPT to GPT-2 and GPT-3.

Section summary

A prompt is a model: measure it on labelled data, with intervals, seeds and paraphrases; ground answers in documents; backtest any risk number.

Videos: the intuition behind the mathematics (in English)

Abbreviations

AIartificial intelligence
APIapplication programming interface
ASEBucharest University of Economic Studies
BERTbidirectional encoder representations from transformers
BM25Best Matching 25 ranking function
BPEbyte-pair encoding
DPOdirect preference optimisation
ECBEuropean Central Bank
ECEexpected calibration error
ESexpected shortfall
EUEuropean Union
FinBERTBERT model adapted to financial text
GARCHgeneralised autoregressive conditional heteroskedasticity
GPTgenerative pre-trained transformer
HShistorical simulation
IPEInstitute for Economic Forecasting, Romanian Academy
KLKullback–Leibler divergence
LLMlarge language model
LoRAlow-rank adaptation
LRlikelihood ratio
MRRmean reciprocal rank
RAGretrieval-augmented generation
RLHFreinforcement learning from human feedback
SFTsupervised fine-tuning
TF-IDFterm frequency–inverse document frequency
VaRValue-at-Risk

References

  • Amza, A., Bolovăneanu, V. & Pele, D. T. (2026). Inflation forecasting with monetary policy narratives: Evidence from an emerging economy. Proceedings of the International Conference on Business Excellence, 20(1), 75–83. doi:10.2478/picbe-2026-0012
  • Brown, T. B. et al. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33. arXiv:2005.14165
  • Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. Q. (2017). On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning, 1321–1330. arXiv:1706.04599
  • Hoffmann, J. et al. (2022). Training compute-optimal large language models. Advances in Neural Information Processing Systems, 35. arXiv:2203.15556
  • Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L. & Chen, W. (2022). LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations. arXiv:2106.09685
  • Kaplan, J. et al. (2020). Scaling laws for neural language models. arXiv:2001.08361
  • Kupiec, P. H. (1995). Techniques for verifying the accuracy of risk measurement models. The Journal of Derivatives, 3(2), 73–84. doi:10.3905/jod.1995.407942
  • Lewis, P. et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33. arXiv:2005.11401
  • Malo, P., Sinha, A., Korhonen, P., Wallenius, J. & Takala, P. (2014). Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65(4), 782–796. doi:10.1002/asi.23062
  • Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35. arXiv:2203.02155
  • Pele, D. T., Bolovăneanu, V., Lin, M.-B., Ren, R., Ginavar, A. T., Spilak, B., Andrei, A.-V., Toma, F.-M., Lessmann, S. & Härdle, W. K. (2026). In the beginning was the Word: LLM-VaR and LLM-ES. Expert Systems with Applications, 295, 128676. doi:10.1016/j.eswa.2025.128676
  • Qwen Team (2024). Qwen2.5 technical report. arXiv:2412.15115
  • Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D. & Finn, C. (2023). Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36. arXiv:2305.18290
  • Robertson, S. & Zaragoza, H. (2009). The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4), 333–389. doi:10.1561/1500000019
  • Sennrich, R., Haddow, B. & Birch, A. (2016). Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the ACL, 1715–1725. doi:10.18653/v1/P16-1162
  • Tak, R. & Pele, D. T. (2025). LLM-driven stock prediction: Capturing market trends with LLaMA. Proceedings of the International Conference on Business Excellence, 19(1), 529–540. doi:10.2478/picbe-2025-0043
  • Wei, J. et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35. arXiv:2201.11903
  • Xiao, S., Liu, Z., Zhang, P. & Muennighoff, N. (2023). C-Pack: Packaged resources to advance general Chinese embedding (BAAI general embedding). arXiv:2309.07597
  • Zhao, Z., Wallace, E., Feng, S., Klein, D. & Singh, S. (2021). Calibrate before use: Improving few-shot performance of language models. Proceedings of the 38th International Conference on Machine Learning, 12697–12706. arXiv:2102.09690

Text written for the web version of the lecture; numbers, charts and sources come from the slides and their Quantlets.