Modelling Financial Markets

Chapter 15: LLMs and Sentiment Analysis

Chapter 15: LLMs and Sentiment Analysis

Daniel Traian PELE

Bucharest University of Economic Studies
IDA Institute Digital Assets
Blockchain Research Center
AI4EFin Artificial Intelligence for Energy Finance
Romanian Academy, Institute for Economic Forecasting
MSCA Digital Finance

I find that high media pessimism predicts downward pressure on market prices followed by a reversion to fundamentals, and unusually high or low pessimism predicts high market trading volume.

— Tetlock (2007), Giving content to investor sentiment: the role of media in the stock market

Tetlock (2007) measures the pessimism of a Wall Street Journal column and shows that it is followed by downward pressure on prices and then by a reversion. The chapter starts from the same question with today's tools: how well do models read the sentiment of a financial text, and can that sentiment be traded? Four routes are compared: word dictionaries, the FinBERT model trained on labelled financial sentences, large language models (LLMs) used zero-shot, and embeddings followed by a classifier. General dictionaries misread financial text: in the lists used here, 77% of the negative words of the Harvard General Inquirer (GI) are not among the negative words of the financial dictionary of Loughran & McDonald (2011).

On financial headlines from Twitter, no model wins outright. The best is the simplest supervised model, TF-IDF (term frequency–inverse document frequency) followed by a logistic regression, with 83.2% accuracy, against 74.2% for Qwen2.5-7B and 72.5% for FinBERT. Labels from the target domain matter more than model size. An LLM also raises a look-ahead bias problem: asked only for the month, Qwen2.5-14B separates up from down months of the S&P 500 (Standard & Poor's) with an area under the ROC (Receiver Operating Characteristic, curve) curve (AUC) of 0.69, where 0.5 means no discrimination.

On 16 large US (United States) stocks, with 106 820 distinct headlines from FNSPID (Financial News and Stock Price Integration Dataset), the scores explain the price move on the day of the news: for FinBERT, one standard deviation of the score corresponds to an excess return 39.2 basis points (bp) higher. Many headlines, however, report the price move itself. A daily long-short strategy on the sign of the score, with positions opened after the news day, brings no net profit. The best score, Qwen2.5-7B, earns 5.64 bp per day before costs, but \(p\) = 0.071 after the Holm correction, and its break-even cost, 1.5 bp per trade, is below the assumed cost of 5 bp. The Sharpe ratio is +0.63 before costs and −1.48 after costs.

Sum of the daily excess returns of the long-short strategy (%), for the FinBERT, Qwen2.5-7B and LM scores, before costs and with 5 bp per trade.
Sum of the daily excess returns of the long-short strategy (%), for the FinBERT, Qwen2.5-7B and LM (Loughran-McDonald, financial sentiment dictionary) scores, before costs and with 5 bp per trade. Interactive chart: “Cumulative Strategy Returns: No Net Profit” →

What you will be able to do

  • Compute the sentiment of a text with a dictionary, with FinBERT and with a zero-shot LLM.
  • Interpret a paired test for two classifiers evaluated on the same texts.
  • Interpret the overlap and memorisation tests of an LLM backtest.
  • Interpret the test of a sentiment signal, with correct timing, inference with few clusters, costs and a correction for multiple testing.
  • Derive the attenuation of the slope on a generated regressor.
  • Interpret prediction-powered inference intervals.

The Further Topics and the Appendix are reading for familiarisation.

Chapter plan

  1. Text as a Source of Financial InformationThe chapter's question, text as data, the definition of sentiment and the first evidence.
  2. DictionariesThe tone of a text measured by word lists: GI, Loughran–McDonald and VADER (Valence Aware Dictionary and sEntiment Reasoner).
  3. Labelled Data and EvaluationThe labelled corpora, accuracy and macro F1, the McNemar test and the comparison of eleven classifiers.
  4. Language Models: FinBERT and LLMsFrom words to BERT, zero-shot classification with Qwen2.5 and model size.
  5. Embeddings as FeaturesGeneric text vectors and the number of labels needed.
  6. Look-Ahead Bias and MemorisationThe memory test on S&P 500 months and Apple days, and the remedies.
  7. From Sentiment to a Trading SignalThe news-day reaction, inference with 16 stocks and the long-short strategy after costs.
  8. Measurement Error in Sentiment ScoresAttenuation of slopes and prediction-powered inference.
  9. Case Studies: ChatGPT and the Colour of Finance WordsThe news reaction read by GPT-4 (Generative Pre-trained Transformer 4) and word lists chosen by returns.
  10. AI for Scientific DiscoveryHypotheses generated by LLMs, the family of tests and the specification curve.
  11. ConclusionsKey ideas and the practical decision.
  12. Further TopicsDetailed examples and advanced topics.
  13. AppendixDerivations, formal tests and readings.

Text as a Source of Financial Information

Prices respond to information, and most of the information that reaches a market arrives as text: news, company filings, earnings calls, analyst reports, central-bank statements and social media. The chapter treats the tone of such text as a quantity to be measured, compares the instruments that measure it on the same labelled data, and then asks whether the measured tone of dated headlines can be traded once timing, costs and multiple testing are taken into account.

The first section fixes the vocabulary: what "text as data" means, which notion of sentiment is used, what the early literature found, and which four families of methods turn a document into a sentiment score.

Can Models Read News Sentiment, and Can We Trade It?

Slide 2 in the PDF
From Text to a Trading Signal
  • Part I: text as data, dictionaries, evaluation of classifiers
    • FinBERT and LLMs (large language models), embeddings
  • Part II: look-ahead bias, the trading signal, scores as estimates
    • Two case studies and one pass of the research loop assisted by AI (artificial intelligence)
  • Detailed examples and advanced topics are in Further Topics, after the Conclusions
What you should be able to do
  • compute the sentiment of a text with a dictionary, FinBERT and a zero-shot LLM (Large Language Model)
  • interpret the paired test that compares two classifiers on the same texts
  • interpret the tests for training overlap and memorised outcomes in an LLM backtest
  • interpret the test of a sentiment signal with correct timing, few-cluster inference, costs and a multiplicity correction
  • derive the attenuation of the slope when a model score is a generated regressor
  • interpret prediction-powered inference intervals
  • Further Topics and Appendix: reading for familiarity

Markets Run on Words

Slide 3 in the PDF

Information that moves prices reaches investors through many channels, and almost all of them are textual: news wires and newspapers, the periodic filings of listed companies, the transcripts of earnings calls, analyst reports, central-bank statements and, more recently, posts on social media. A model of how prices respond to news therefore needs, at some stage, a way of reading text.

Financial newspapers are as old as organised markets. The Wall Street Journal published its first issue on 8 July 1889, and the photograph on the slide shows that first page. The difference today is one of scale: millions of documents appear every day, far more than any analyst can read. Reading at this scale has to be delegated to an algorithm, and the quality of that algorithm becomes a measurement problem in its own right.

In brief

  • Prices move when information arrives, and most information arrives as text
    • News, company filings, earnings calls, analyst reports, central-bank statements, social media
  • Financial newspapers are as old as organised markets
    • The Wall Street Journal: first issue on 8 July 1889
  • Today: millions of documents per day, far more than any analyst can read
Markets Run on Words The Wall Street Journal, first issue, 8 July 1889 Photo: Dow Jones & Company (1889); public domain; Wikimedia Commons

Text as Data, and What We Mean by Sentiment

Slide 4 in the PDF

Gentzkow, Kelly & Taddy (2019) describe the general recipe of text as data in economics. A document \(d\) is first mapped to a numerical representation \(x_d\): word counts, scores or vectors. This representation is then mapped to a quantity of interest, such as sentiment, topic or risk, and that quantity enters an ordinary statistical model of an economic outcome, such as returns or volatility. Every step involves choices, and every choice can introduce error.

In this chapter the sentiment of a text is a property of the text: whether it is good, bad or neutral news for the asset it mentions. It is reported either as one of three classes or as a net score

\[s = P(\text{positive}) - P(\text{negative}) \in [-1, 1],\]
(15.1)

where the two probabilities are those a classifier assigns to the positive and negative classes. This notion is distinct from investor sentiment, the mood of investors measured from market data (Baker & Wurgler, 2006). The surveys of Kearney & Liu (2014) and Loughran & McDonald (2016) cover both strands of the finance literature.

The chapter treats the score (15.1) as an estimate of the true tone of a text, measured with error. This view runs through all the sections: it determines how classifiers are evaluated against human labels, how regression slopes on scores are read, and how confidence intervals are built when model scores replace human labels.

In brief

  • Gentzkow, Kelly & Taddy (2019): a document \(d\) becomes numbers \(x_d\) (counts, scores, vectors)
    • These numbers feed an ordinary statistical model
    • Map \(x_d\) to a quantity of interest (sentiment, topic, risk), then use it in an economic question (returns, volatility)
  • Sentiment of a text: good, bad or neutral news for the asset it mentions
    • Three classes, or the net score \(s = P(\text{positive}) - P(\text{negative}) \in [-1, 1]\)
    • Not investor sentiment, the mood of investors measured from market data (Baker & Wurgler (2006))
  • Surveys for finance: Kearney & Liu (2014), Loughran & McDonald (2016)
  • This chapter: the score is an estimate of the true tone, with measurement error

Early Evidence and Narratives

Slide 5 in the PDF

Four contributions frame the empirical question. Antweiler & Frank (2004) study internet message-board posts about stocks and find that they help predict volatility, while their effect on returns is small. Tetlock (2007) measures pessimism in a Wall Street Journal column and finds that high media pessimism predicts downward pressure on prices, followed by a reversal; unusually high or low pessimism also predicts high trading volume.

Tetlock, Saar-Tsechansky & Macskassy (2008) move from the market to individual firms. Their measure is the share of negative words in firm-specific news, and it predicts lower earnings and lower returns. Prices incorporate this information only gradually, which is the kind of slow adjustment a trading strategy could exploit.

Shiller (2017) gives the broader motivation: popular stories spread like epidemics and influence economic decisions, and they can be measured through text. Shiller received the Nobel Prize in Economics in 2013, together with Fama and Hansen. Narratives can push prices away from fundamentals, a theme treated in Chapter 2 on market efficiency and Chapter 17 on bubbles and crashes.

In brief

  • Antweiler & Frank (2004): internet message-board posts about stocks help predict volatility
    • The effect on returns is small
  • Tetlock (2007): high media pessimism predicts downward pressure on prices
    • Pessimism measured in a Wall Street Journal column
    • The pressure is followed by a reversal
    • Unusually high or low pessimism also predicts high trading volume
  • Tetlock, Saar-Tsechansky & Macskassy (2008): negative words in firm-specific news predict lower earnings and returns
    • Measure: the share of negative words
    • Prices incorporate this information only gradually
  • Shiller (2017): popular stories spread like epidemics and move economic decisions
    • They can be measured through text
    • Shiller: Nobel Prize in Economics 2013, with Fama and Hansen; narratives can push prices away from fundamentals (Chapters 2 and 17)
Early Evidence and Narratives Robert J. Shiller, Stockholm, December 2013 Photo: Bengt Nyman (2013); CC BY 2.0; Wikimedia Commons

Four Ways to Turn Text into Sentiment

Slide 6 in the PDF

The methods compared in the chapter fall into four families, which differ in how much labelled data they need. A dictionary counts the words of a document that appear on predefined positive and negative lists; it needs no labels. Three dictionaries are used:

  • the Harvard General Inquirer (GI), a general-purpose word list;
  • the Loughran–McDonald (LM) financial dictionary;
  • VADER (Valence Aware Dictionary and sEntiment Reasoner), a dictionary with rules built for social media.

A fine-tuned model such as FinBERT is a neural network trained on thousands of labelled sentences. A large language model (LLM), such as Qwen2.5 or the GPT (Generative Pre-trained Transformer) models of OpenAI, receives an instruction in plain language, the prompt, and classifies without any labelled example; this is zero-shot use. Finally, embeddings followed by a classifier turn each text into a generic vector, here from the MiniLM model, and fit a regression on hundreds to thousands of labels.

All four are compared on the same labelled data, with 95% confidence intervals (CI), and then tested on daily stock returns. A fifth route, text regressions with statistical theory (Taddy, 2013; Ke, Kelly & Xiu, 2019), is not tested in the body of the chapter; it is described in the Appendix.

In brief

MethodHow it worksLabels neededExamples
Dictionarycount listed wordsnoneGI, LM, VADER
Fine-tuned modelnetwork trained on labelled sentencesthousandsFinBERT
Large language modelinstruction in plain language (prompt)none (zero-shot)Qwen2.5, GPT
Embeddings + classifiertext vectors into a regressionhundreds to thousandsMiniLM
GI: the Harvard General Inquirer, a general-purpose word list; LM (Loughran-McDonald, financial sentiment dictionary): the Loughran–McDonald financial dictionary; VADER: Valence Aware Dictionary and sEntiment Reasoner, a dictionary with rules built for social media; GPT: Generative Pre-trained Transformer, the language models from OpenAI
  • We compare all four on the same labelled data, with 95% confidence intervals (CI)
  • Then we test whether any of them makes money on daily stock returns
  • A fifth route, text regressions with statistical theory, is not tested here (Appendix)
    • Taddy; Ke, Kelly and Xiu

Section summary

Text enters finance as data: a document is mapped to numbers, the numbers to a sentiment score, and the score to a model of returns. The early literature found that the tone of news is related to prices, volume and earnings, with slow adjustment. The four families of methods differ mainly in the amount of labelled data they need, and the following sections compare them on the same texts.

Dictionaries

The oldest method in finance measures tone by counting words from predefined lists. Its appeal is transparency: every score can be traced to the words that produced it. Its weakness is that a word list ignores context, and that a list built for one domain can misread another.

Can a Word List Measure the Tone of a Financial Text?

Slide 7 in the PDF

A dictionary method rests on the bag-of-words representation: a document is reduced to the counts of its words, and their order is ignored. Tokens are case-insensitive words, without digits and punctuation. A dictionary supplies a set \(\mathcal{P}\) of positive words and a set \(\mathcal{N}\) of negative words; let \(P_d\) and \(N_d\) be the numbers of positive and negative words in document \(d\).

The tone of the document is

\[\tau_d = \frac{P_d - N_d}{P_d + N_d} \in [-1, 1],\]
(15.2)

with \(\tau_d = 0\) if \(P_d + N_d = 0\), and the predicted class is the sign of \(\tau_d\), neutral when \(\tau_d = 0\). A document with only positive listed words has tone \(+1\), one with only negative words has tone \(-1\), and a document with no listed word is neutral by construction.

The method is transparent, fast and needs no training data. Its limitation follows directly from (15.2): the sign of each word is fixed in advance, so the method is blind to context and to negation. "Not good" counts as positive, and a word whose meaning depends on its neighbours receives the same sign in every sentence.

In brief

  • Bag of words: a document is reduced to its word counts
    • Word order is ignored
    • Tokens: case-insensitive words, without digits and punctuation
  • A dictionary lists positive words \(\mathcal{P}\) and negative words \(\mathcal{N}\)
    • \(P_d\), \(N_d\): number of positive and negative words in document \(d\)
    • Tone: \(\tau_d = \dfrac{P_d - N_d}{P_d + N_d} \in [-1, 1]\), and \(\tau_d = 0\) if \(P_d + N_d = 0\); the class is the sign of \(\tau_d\) (neutral for \(\tau_d = 0\))
  • Transparent, fast and free of training data
    • Blind to context and negation

Generic, Financial and Rule-Based Dictionaries

Slide 8 in the PDF

The Harvard General Inquirer (GI), word list Harvard IV-4, is a psychology dictionary from the 1960s, with 2,007 negative and 1,637 positive words. Loughran & McDonald (2011) built the LM dictionary from the annual reports (10-K filings) of US firms. It has 2,355 negative words and only 354 positive words. Their finding was that almost three quarters of the GI negative words are not negative in a financial context. In the lists used in this chapter, 1,552 GI negative words, 77% of them, are not LM negative words.

VADER (Hutto & Gilbert, 2014) contains about 7,500 terms with intensity scores and was built for social media. It adds rules for negation ("not good"), intensifiers ("very"), capital letters, exclamation marks and emoticons, and it returns a compound score in \([-1, 1]\), usually cut at \(\pm 0.05\) to obtain three classes.

The three dictionaries thus differ along two axes: the vocabulary (generic for GI and VADER, financial for LM) and the treatment of context (none for GI and LM, simple rules for VADER). The rules of VADER fix some context problems, but its vocabulary remains generic, like that of GI.

In brief

  • Harvard General Inquirer (GI), word list Harvard IV-4: a psychology dictionary from the 1960s
    • 2 007 negative and 1 637 positive words
  • Loughran & McDonald (2011) (LM): built from annual reports (10-K filings) of US firms
    • 2 355 negative and only 354 positive words
    • Their finding: almost three quarters of the GI negative words are not negative in a financial context
    • In the lists used here, 1 552 GI negative words (77%) are not LM negative words
  • VADER (Hutto & Gilbert (2014)): about 7,500 terms with intensity scores, built for social media
    • Rules for negation (“not good”), intensifiers (“very”), capital letters, exclamation marks and emoticons
    • Compound score in \([-1, 1]\), usual cut-offs \(\pm 0.05\)
    • The rules fix some context problems, but the vocabulary is generic, like GI

Negative for a Psychologist, Neutral for an Accountant (1/2)

Slide 9 in the PDF

The chart examines the words on which GI and LM disagree, using the sentences of Financial PhraseBank, a corpus of news sentences labelled by human annotators and described in the next section. Each horizontal bar is a word; its length is the number of PhraseBank sentences that contain the word, and its colours split those sentences by the label the annotators gave them: positive, neutral or negative.

In the interactive version, the selector chooses the word list: words negative in GI but not in LM, or the reverse. The slider sets how many words are shown, and the ▶ button adds words one at a time. The readout below the chart counts the sentences and the share of them not labelled negative.

    Question for the room "Service", "capital" and "board" are negative words in the Harvard IV-4 dictionary. In Financial PhraseBank, what proportion of the sentences containing each of these words is labelled negative?
    Answer

    Below 10% for each word: 2 of the 106 sentences with "service" (1.9%), 7 of the 104 with "capital" (6.7%) and 3 of the 79 with "board" (3.8%). The sign of a word in the dictionary does not determine the label of the sentence. Nor would a word unrelated to tone give 50%: it would occur in negative sentences at the frequency of the negative class in the corpus, 12.5%.

    For the default list, the most frequent words, "service" (106 sentences), "capital" (104) and "board" (79), are business nouns. Only 2 of the 106 sentences with "service" (1.9%), 7 of the 104 with "capital" (6.7%) and 3 of the 79 with "board" (3.8%) are labelled negative. A word unrelated to tone would occur in negative sentences at the frequency of the negative class in the corpus, 12.5%; these words fall below even that.

    Negative for a Psychologist, Neutral for an Accountant (2/2)

    Slide 10 in the PDF

    The words that GI marks as negative and LM does not are, for the most part, ordinary vocabulary of business. "Service", "capital" and "board" appear in sentences that are not labelled negative in 98%, 93% and 96% of cases, and "vice", as in "vice president", in 100% of cases. Counting them as negative adds noise to the tone of every financial text that mentions them.

    "Lower" is a different case: only 26% of its sentences are not labelled negative, so it is often a genuinely negative word. Yet "lower costs" is good news for a firm. The sign of "lower" depends on the next word, which no word list sees; this is the context problem of (15.2) in its purest form.

    Taken together, the 964 PhraseBank sentences whose only GI-negative words are missing from the LM negative list are not labelled negative in 89% of cases. A worked example of one sentence scored by both dictionaries is in Further Topics.

    In brief

    • Words negative in GI but not in LM
      • Ranked by the number of Financial PhraseBank sentences that contain them
      • Colour: the label human annotators gave the sentence
    • “Service” (106 sentences), “capital” (104) and “board” (79) are business nouns, not bad news
      • Not labelled negative: 98%, 93%, 96% of their sentences; “vice” (“vice president”): 100%
    • “Lower”: only 26% not negative, but “lower costs” is good news
      • The sign depends on the next word, which no word list sees
    • Of the 964 sentences whose only GI-negative words are missing from the LM negative list:
      • 89% are not labelled negative
    MFM_ch15_dictionaries

    Section summary

    Dictionary tone (15.2) is transparent and cheap, but its quality depends on the list. A generic psychology dictionary counts business nouns as negative: in the lists used here, 77% of the GI negative words are not negative in LM, and sentences scored negative only by such words are not labelled negative in 89% of cases. No list sees context, so words such as "lower" keep the wrong sign whenever the next word reverses it.

    Labelled Data and Evaluation

    A classifier is judged by its agreement with human labels on texts it has not been trained on. This section describes the two labelled corpora, the metrics used when one class dominates, the paired test that compares two classifiers on the same texts, and the comparison of eleven classifiers.

    Which Texts Do We Judge a Classifier On? Two Labelled Corpora

    Slide 11 in the PDF

    Financial PhraseBank (Malo et al., 2014) contains 4,846 sentences from financial news. They were labelled by a pool of 16 annotators with a finance background, 5 to 8 per sentence, who answered one question: is the sentence good, bad or neutral news for the investor? The corpus is released with agreement levels, the share of a sentence's annotators that chose the majority label: at least 50%, 66%, 75% or 100%. Full agreement is reached on 2,264 sentences, and 629 sentences have agreement below 66%. The labels are 59% neutral, 28% positive and 12% negative.

    The second corpus, Twitter Financial News, is distributed on the Hugging Face Hub under the MIT (Massachusetts Institute of Technology) licence and contains financial headlines from 2022, labelled bearish, bullish or neutral. It has a training set of 9,543 headlines and a validation set of 2,388 headlines, which serves for evaluation throughout the chapter. Its labels are 66% neutral, 20% positive and 15% negative. The texts are short and noisy, with tickers and links, and are closer to the real news flow than the edited sentences of PhraseBank.

    Two corpora are used because a result obtained on one need not carry over to the other. The comparison matters in particular for FinBERT, whose public version was fine-tuned on PhraseBank, as the section on language models shows.

    In brief

    • Financial PhraseBank (Malo et al. (2014)): 4 846 news sentences
      • Labelled by a pool of 16 annotators with a finance background, 5 to 8 per sentence
      • Question: good, bad or neutral news for the investor?
      • Agreement levels: at least 50%, 66%, 75% or 100% of the sentence's annotators; 2 264 sentences with full agreement, 629 below 66%
      • Labels: 59% neutral, 28% positive, 12% negative
    • Twitter Financial News (Hugging Face Hub; MIT licence): headlines from 2022
      • Labelled bearish, bullish or neutral
      • Training set of 9,543, validation set of 2 388 headlines (used for evaluation)
      • Labels: 66% neutral, 20% positive, 15% negative
      • Short, noisy texts with tickers and links: closer to the real news flow
    • Two corpora, because a result on one may not carry over to the other

    Accuracy or Macro-F1? Metrics for Unbalanced Classes

    Slide 12 in the PDF

    Accuracy is the share of texts classified correctly, \(\text{acc} = \frac{1}{n}\sum_{i=1}^{n}\mathbf{1}\{\hat y_i = y_i\}\). When one class dominates, accuracy rewards a classifier that predicts that class most of the time. A metric that weighs the classes equally is needed to see whether the rare classes, here positive and negative news, are recognised at all.

    For class \(k\), let \(TP_k\) be the number of texts of class \(k\) predicted as \(k\), \(FP_k\) the number of texts of other classes predicted as \(k\), and \(FN_k\) the number of texts of class \(k\) predicted as another class. Precision, recall and their harmonic mean are

    \[\begin{gathered} P_k = \frac{TP_k}{TP_k + FP_k}, \qquad R_k = \frac{TP_k}{TP_k + FN_k}, \\ F1_k = \frac{2P_kR_k}{P_k + R_k}, \end{gathered}\]

    and the macro-F1 is their average over the classes,

    \[\text{macro-F1} = \frac{1}{K}\sum_{k=1}^{K} F1_k,\]
    (15.3)

    with \(K = 3\) classes. Macro-F1 gives every class the same weight, however rare; the same metrics are used in Chapter 13.

    The benchmark that always answers "neutral" shows the difference between the two metrics. On the Twitter validation set it reaches 66% accuracy, the share of neutral headlines. Its neutral recall is 1 and its neutral precision 0.656, so \(F1_{\text{neutral}} = 2 \times 0.656 \times 1/(0.656 + 1)\), that is 0.792, while \(F1 = 0\) for the two other classes. Its macro-F1 is therefore \(0.792/3\), only 26.4%.

    In brief

    • The question: how do we compare classifiers when one class dominates?
    • Accuracy: \(\text{acc} = \frac{1}{n}\sum_{i=1}^{n} \mathbf{1}\{\hat y_i = y_i\}\)
    • For class \(k\): precision \(P_k = \dfrac{TP_k}{TP_k + FP_k}\), recall \(R_k = \dfrac{TP_k}{TP_k + FN_k}\), \(F1_k = \dfrac{2 P_k R_k}{P_k + R_k}\)
      • \(TP_k\): texts of class \(k\) predicted \(k\); \(FP_k\): texts of other classes predicted \(k\); \(FN_k\): texts of class \(k\) predicted otherwise
    • Macro-F1 \(= \frac{1}{K}\sum_{k=1}^{K} F1_k\), with \(K = 3\)
      • Every class counts the same, however rare (Chapter 13)
    • Benchmark “always neutral” on Twitter: 66% accuracy, but a macro-F1 of only 26.4%
      • \(F1_{\text{neutral}} = 2 \times 0.656 \times 1/(0.656 + 1) = 0.792\), and \(F1 = 0\) for the other two classes, so macro-F1 \(= 0.792/3\)

    When Is One Classifier Better? The McNemar Test on the Same Texts

    Slide 13 in the PDF

    When two classifiers A and B are evaluated on the same \(n\) texts, their accuracies are correlated: both find the easy texts easy and the hard texts hard. Two separate confidence intervals therefore do not answer whether one classifier is better. The comparison has to use the pairing.

    Let \(n_{01}\) be the number of texts on which A is wrong and B is right, and \(n_{10}\) the number on which A is right and B is wrong. Texts on which the two classifiers agree, both right or both wrong, carry no information about the difference: the accuracy gap is \((n_{01} - n_{10})/n\). Under the null hypothesis of equal accuracy, each discordant text is equally likely to favour either classifier, so \(n_{01} \sim \text{Binomial}(n_{01} + n_{10}, 0.5)\), which gives an exact test. For large \(n_{01} + n_{10}\) the statistic

    \[\chi^2 = \frac{(|n_{01} - n_{10}| - 1)^2}{n_{01} + n_{10}}\]
    (15.4)

    is approximately \(\chi^2_1\); the \(-1\) is a continuity correction.

    A small example shows the exact version. If A is wrong and B right on 4 texts and the reverse happens on 1, then with \(X \sim \text{Binomial}(5, 0.5)\) the two-sided p-value is \(p = 2\,P(X \ge 4) = 2 \times 6/32 = 0.375\). Five discordant texts are far too few to separate the two classifiers. A worked example on real data, FinBERT against LM, is in the Appendix.

    In brief

    • Two classifiers A and B on the same \(n\) texts: their accuracies are correlated
      • So two separate CIs do not answer the question
    • \(n_{01}\): number of texts where A is wrong and B is right
    • \(n_{10}\): number of texts where A is right and B is wrong
    • Texts on which both classifiers agree carry no information
      • The accuracy gap is \((n_{01} - n_{10})/n\)
    • Under \(H_0\) (equal accuracy), \(n_{01} \sim \text{Binomial}(n_{01} + n_{10}, 0.5)\)
      • An exact test (worked example: Appendix)
    • For large \(n_{01} + n_{10}\): \(\chi^2 = (|n_{01} - n_{10}| - 1)^2/(n_{01} + n_{10})\) is approximately \(\chi^2_1\)
      • \(-1\): continuity correction
    • Example: A is wrong and B right on 4 texts, the reverse on 1
      • So \(p = 2\,P(X \ge 4) = 2 \times 6/32 = 0.375\), with \(X \sim \text{Binomial}(5, 0.5)\)

    Accuracy of Eleven Classifiers on Two Data Sets (1/2)

    Slide 14 in the PDF

    The chart compares eleven classifiers on the two corpora. Each row is a method: the three dictionaries (GI, VADER, LM), two supervised models trained on Twitter labels, TF-IDF + LR (Logistic Regression) and MiniLM + LR, FinBERT and five zero-shot Qwen2.5 models from 0.5 to 14 billion parameters. TF-IDF (term frequency–inverse document frequency) weights word counts down for words common to many texts, and LR is logistic regression. The horizontal axis is the accuracy in percent with its 95% CI; the left panel is Financial PhraseBank and the right panel Twitter Financial News. The dashed vertical line marks the "always neutral" benchmark.

    In the interactive version, the first selector switches between accuracy and macro-F1, and the slider restricts PhraseBank to sentences with higher annotator agreement: all sentences, at least 66%, at least 75% or 100%. The ▶ button raises the agreement level step by step. The readouts report the values of FinBERT, Qwen2.5-7B and the benchmark on PhraseBank, and the best method and the dictionaries on Twitter.

      Question for the room On the Twitter headlines, which of the 11 methods has the highest accuracy?
      Answer

      TF-IDF + logistic regression, with 83.2%. FinBERT has 72.5%, zero-shot Qwen2.5-7B 74.2% and Qwen2.5-14B 72.4%: neither domain adaptation nor parameter count guarantees first place. All three dictionaries stay below the "always neutral" benchmark (65.6%). TF-IDF + LR is trained on Twitter labels, so the comparison does not isolate the effect of the architecture.

      On Twitter, the highest accuracy, 83.2%, belongs to TF-IDF + logistic regression. Zero-shot Qwen2.5-7B reaches 74.2%, FinBERT 72.5% and Qwen2.5-14B 72.4%: neither adaptation to financial text nor the number of parameters guarantees first place. All three dictionaries stay below the "always neutral" benchmark, 65.6%. Because TF-IDF + LR is the only method of this kind trained on Twitter labels, the comparison does not isolate the effect of the architecture.

      In brief

      TF-IDF: term frequency–inverse document frequency, word counts weighted down for common words; LR: logistic regression MFM_ch15_classification

      Accuracy of Eleven Classifiers on Two Data Sets (2/2)

      Slide 15 in the PDF

      Two of the eleven methods, TF-IDF + LR and MiniLM + LR, are trained on the Twitter training set; all others see no Twitter label. This difference in training data explains most of the ranking on Twitter. On that corpus, all three dictionaries lose to the majority class: GI reaches 43.6%, VADER 49.6% and LM 60.5%, against 66% for "always neutral".

      FinBERT reaches 89.0% on PhraseBank and 72.5% on Twitter; zero-shot Qwen2.5-7B reaches 81.4% and 74.2%. The best result on Twitter, 83.2%, comes from the simplest supervised model, TF-IDF + logistic regression. The lesson is that labelled data from the target domain beat model size.

      The comparison is one of procedures on two given corpora: it ranks complete pipelines, including their training data, and does not attribute the differences to any single component. Human disagreement and Cohen's \(\kappa\), which corrects agreement for chance, are discussed in Further Topics.

      In brief

      • TF-IDF + LR and MiniLM + LR: the only methods trained on Twitter labels
      • All three dictionaries lose to the majority class on Twitter
        • Twitter: GI 43.6%, VADER 49.6%, LM 60.5%; “always neutral”: 66%
      • FinBERT: 89.0% on PhraseBank, 72.5% on Twitter; Qwen2.5-7B zero-shot: 81.4% and 74.2%
      • Best on Twitter, 83.2%: the simplest supervised model, TF-IDF + logistic regression
      • Lesson: labelled data from the target domain beat model size
      • Type of evidence: comparison of procedures (it shows which method works better on the same data, not why)
      MFM_ch15_classification

      Section summary

      Classifiers are compared on two labelled corpora, with accuracy and macro-F1 (15.3), and pairs of classifiers with the McNemar test (15.4), which uses only the texts on which they disagree. On the Twitter headlines, the dictionaries do worse than always answering "neutral", FinBERT and zero-shot Qwen2.5-7B are close (72.5% and 74.2%), and a TF-IDF logistic regression trained on Twitter labels leads with 83.2%.

      Language Models: FinBERT and LLMs

      Dictionaries give each word a fixed sign. Language models read each word in the context of the sentence. This section describes the fine-tuned model FinBERT, the trap of evaluating it on its own training data, the step from next-word prediction to instruction-following LLMs, and the zero-shot classification of headlines with the Qwen2.5 family.

      Does Context Help? From Words to BERT

      Slide 16 in the PDF

      A dictionary gives "lower" one fixed sign. A language model instead represents each token by a vector that depends on all the other tokens of the sentence, through the attention mechanism of the Transformer (Vaswani et al., 2017), presented in Chapter 14. The representation of "lower" in "lower costs" can then differ from its representation in "lower revenue".

      BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al., 2019) is a Transformer encoder pretrained on books and Wikipedia by guessing words hidden at random. It has about 110 million parameters. For classification it uses the final vector \(h_{\text{CLS}} \in \mathbb{R}^{768}\) of a special token, [CLS (classification token of BERT)], placed at the start of every text.

      Fine-tuning adds a classification layer on top of \(h_{\text{CLS}}\) and trains the whole network on labelled sentences. The class probabilities and the training loss are

      \[p = \operatorname{softmax}(W h_{\text{CLS}} + c), \qquad W \in \mathbb{R}^{3\times 768},\]
      (15.5)
      \[\mathcal{L} = -\frac{1}{n}\sum_{i=1}^{n} \ln p_{i,y_i},\]

      where \(p\) contains the probabilities of the negative, neutral and positive classes, \(y_i\) is the human label of sentence \(i\) and \(\mathcal{L}\) is the cross-entropy. Minimising \(\mathcal{L}\) adjusts both the new layer and the pretrained weights.

      In brief

      • A dictionary gives “lower” one fixed sign
      • BERT (Bidirectional Encoder Representations from Transformers; Devlin et al. (2019))
        • A Transformer encoder pretrained on books and Wikipedia by guessing words hidden at random
        • About 110 million parameters
        • For classification it uses the final vector \(h_{\text{CLS}} \in \mathbb{R}^{768}\) of the special token [CLS (classification token of BERT)] (for classification), placed at the start of every text
      • Fine-tuning: a classification layer on top of \(h_{\text{CLS}}\)
        • The whole network is trained on labelled sentences
        • \(p = \operatorname{softmax}(W h_{\text{CLS}} + c)\), with \(W \in \mathbb{R}^{3 \times 768}\): probabilities of the negative, neutral and positive classes
        • Training minimises the cross-entropy \(-\frac{1}{n}\sum_i \ln p_{i, y_i}\), with \(y_i\) the human label of sentence \(i\)

      FinBERT: BERT Adapted to Financial Text

      Slide 17 in the PDF

      FinBERT (Araci, 2019) is BERT further pretrained on financial news and then fine-tuned for sentiment classification. The public model used in this chapter, ProsusAI/finbert, was fine-tuned on Financial PhraseBank. It takes a few milliseconds per sentence on a laptop, which makes it practical for large news archives.

      The score used throughout the chapter is the net score (15.1), \(s = p_{\text{pos}} - p_{\text{neg}} \in [-1, 1]\), computed from the probabilities in (15.5), and the predicted label is \(\arg\max_k p_k\). FinBERT is the benchmark because it is public, fast and widely used in the finance literature.

      Other versions of the model exist. Huang, Wang & Yang (2023) train a FinBERT on analyst reports, earnings-call transcripts and 10-K filings, and find that it outperforms dictionaries, mainly on sentences that dictionaries misread. The error patterns of FinBERT on the Twitter headlines, compared with those of LM and Qwen2.5-7B, are in Further Topics.

      In brief

      • Araci (2019): FinBERT is BERT further pretrained on financial news, then fine-tuned for sentiment
        • The public model used here (ProsusAI/finbert) was fine-tuned on Financial PhraseBank
        • A few milliseconds per sentence on a laptop
      • The score used in this chapter: \(s = p_{\text{pos}} - p_{\text{neg}} \in [-1, 1]\)
        • The label is \(\arg\max_k p_k\)
      • Why FinBERT as the benchmark: public, fast and widely used in the finance literature
      • Huang, Wang & Yang (2023): a FinBERT trained on analyst reports, earnings-call transcripts and 10-K filings
        • It outperforms dictionaries, mainly on sentences that dictionaries misread

      In-Sample Evaluation Bias

      Slide 18 in the PDF

      FinBERT reaches 89.0% accuracy on PhraseBank, but most of PhraseBank was used to train it: Araci (2019) used 3,101 of the 4,845 sentences. On the Twitter headlines its accuracy is 72.5%, a drop of 16.4 points. Part of this drop is genuine difficulty: Qwen2.5-7B, which was not fine-tuned on these labels, also drops, by 7.2 points, because Twitter headlines are harder.

      The excess drop of FinBERT is measured by a difference in differences,

      \[\Delta = (a_{F,P} - a_{F,T}) - (a_{Q,P} - a_{Q,T}),\]
      (15.6)

      which equals 9.2 points, where \(a_{m,D}\) is the accuracy of model \(m\) (F for FinBERT, Q for Qwen) on data \(D\) (P for PhraseBank, T for Twitter). Its bootstrap 95% CI is [6.7; 11.9]. The identifying assumption is that, without training overlap, both models would lose the same amount from PhraseBank to Twitter. The interval measures sampling noise only, not this assumption: the two architectures may react differently to a new domain.

      The result supports a descriptive difference between the two accuracy drops. Attributing \(\Delta\) to training overlap would also require an uncontaminated comparator and no other differential change; Qwen was not fine-tuned on PhraseBank, but the texts may be in its pretraining data. The practical rule is the one of the out-of-sample tests of Chapters 8 and 13: a language model is evaluated only on texts it has never seen.

      In brief

      • FinBERT: 89.0% on PhraseBank, most of which it was trained on
        • Araci (2019): 3,101 of the 4,845 sentences in his version (4 846 here)
        • 72.5% on Twitter: a drop of 16.4 points (from unrounded values)
        • Qwen2.5-7B, not fine-tuned on these labels, also drops by 7.2 points: Twitter headlines are harder
      • Difference in differences: \(\Delta = (a_{F,P} - a_{F,T}) - (a_{Q,P} - a_{Q,T})\) = 9.2 points
        • Bootstrap CI 95% [6.7; 11.9]
        • \(a_{m,D}\): accuracy of model \(m\) (F: FinBERT, Q: Qwen) on data \(D\) (P: PhraseBank, T: Twitter)
        • Identifying assumption: without training overlap, both models would lose the same amount from PhraseBank to Twitter
        • The CI measures sampling noise, not the assumption (the two architectures may react differently to a new domain)
      • Type of evidence: a descriptive difference between the accuracy drops
        • Attributing \(\Delta\) to training overlap also needs an uncontaminated comparator and no other differential change
        • Qwen was not fine-tuned on PhraseBank, but the texts may be in its pretraining data
      • Rule: evaluate a language model only on texts it has never seen

      From Next-Word Prediction to Instructions

      Slide 19 in the PDF

      Shannon (1951) showed that English is highly predictable from its past letters. A large language model (LLM) builds on the same idea: it estimates the conditional distribution \(P(w_{t+1} \mid w_1, \dots, w_t)\) of the next token, where \(w_t\) is the \(t\)-th token, a word or part of a word. Trained on enough text, this single objective produces models that can complete a wide range of tasks.

      Brown et al. (2020) showed that, at large scale, such models solve tasks from a few examples placed in the prompt, which is few-shot use. Ouyang et al. (2022) added instruction tuning and RLHF (reinforcement learning from human feedback), which teach the model to follow requests written in plain language. The zero-shot classification used in this chapter relies on this ability.

      Domain-specific LLMs also exist. BloombergGPT (Wu et al., 2023) has 50 billion parameters and was trained on financial and general text. Background on next-word prediction and natural language processing is available in the Quantinar courses linked on the slide.

      In brief

      From Next-Word Prediction to Instructions Claude E. Shannon, founder of information theory Photo: Tekniska museet, Stockholm (undated); CC BY 2.0; Wikimedia Commons

      Zero-Shot Classification with Qwen2.5: the Prompt and the Probabilities

      Slide 20 in the PDF

      The zero-shot classifier needs no labelled example. The prompt consists of a system instruction, "You classify the sentiment of financial news for investors. Answer with one word: positive, negative or neutral.", followed by the text in the form "Headline: <text> Sentiment:". The model's answer is not generated as free text; instead, the probabilities of the three answer words are read directly.

      At the first generated position, let \(z_k\) be the logits of the tokens "positive", "negative" and "neutral". Normalising over these three words gives

      \[p_k = \frac{e^{z_k}}{\sum_j e^{z_j}}, \qquad k \in \{\text{pos}, \text{neg}, \text{neu}\}.\]
      (15.7)

      The label is \(\arg\max_k p_k\), and the score is \(p_{\text{pos}} - p_{\text{neg}}\), as for FinBERT. Reading probabilities rather than parsing generated text makes the classifier deterministic and gives a continuous score that can enter a regression.

      In brief

      • Prompt: a system instruction plus the text to classify
        • You classify the sentiment of financial news for investors. Answer with one word: positive, negative or neutral.
        • Headline: <text> Sentiment:
      • We read the logits \(z_k\) (the model's unnormalised scores) of the tokens “positive”, “negative”, “neutral” at the first generated position
        • \(p_k = e^{z_k} / \sum_j e^{z_j}\), normalised over the three answer words
        • Label: \(\arg\max_k p_k\)
        • Score: \(p_{\text{pos}} - p_{\text{neg}}\), as for FinBERT

      Temperature and the Qwen2.5 Models

      Slide 21 in the PDF

      When an LLM generates text, it draws the next token from a distribution controlled by the temperature \(T > 0\):

      \[p_k(T) = \frac{e^{z_k/T}}{\sum_j e^{z_j/T}}.\]
      (15.8)

      As \(T \to 0\) the model always chooses the most likely token (greedy decoding) and gives the same answer on every run; Lopez-Lira & Tang (2026) use temperature 0. At \(T = 1\) the model samples from its own distribution, and \(T > 1\) flattens the probabilities, so the answers vary more.

      No text is generated in this chapter. The label \(\arg\max_k p_k\) is the answer the model would give as \(T \to 0\), and the score keeps the \(T = 1\) probabilities of (15.7), with nothing to parse. The models are from the Qwen2.5 family (Qwen Team, 2024), open-weight models from 0.5 to 72 billion parameters released on 19 September 2024 under the Apache 2.0 licence, or the Qwen Research License for 3B. The chapter uses 0.5B, 1.5B, 3B, 7B and 14B, where B stands for billion parameters; their processing times are in the Appendix.

      Open weights matter for two reasons:

      1. Results can be reproduced with exactly the same model.
      2. The release date of the weights bounds what the model could know, a property used in the section on look-ahead bias.

      In brief

      • When generating, the model draws the next token from \(p_k(T) = e^{z_k/T} / \sum_j e^{z_j/T}\)
        • With temperature \(T > 0\)
        • \(T \to 0\): always the most likely token (greedy decoding), so the same answer on every run; Lopez-Lira & Tang (2026) use temperature 0
        • \(T = 1\): the model's own distribution; \(T > 1\) flattens the probabilities, so answers vary more
      • No text is generated here: the label \(\arg\max_k p_k\) is the \(T \to 0\) answer
        • The score keeps the \(T = 1\) probabilities, with nothing to parse
      • Qwen Team (2024): open-weight models, from 0.5 to 72 billion parameters
        • Licences: Apache 2.0; Qwen Research License for 3B
        • Released on 19 September 2024
      • We use 0.5B, 1.5B, 3B, 7B and 14B (B: billion parameters)
      • Open weights: results can be reproduced
        • The release date bounds what the model could know

      Model Size and Zero-Shot Accuracy

      Slide 22 in the PDF

      The chart shows the macro-F1 of the five zero-shot Qwen2.5 models, all with the same prompt. The horizontal axis is the number of parameters in billions, on a log scale, and the vertical axis is macro-F1 in percent. The blue line is Financial PhraseBank and the red line Twitter Financial News; the dotted horizontal lines are FinBERT on the two corpora.

      In the interactive version, the selector switches between macro-F1 and accuracy, and the slider chooses the prompt used on Twitter, P1, P2 or P3; the three prompts ask the same question in different words. The ▶ button runs through the three prompts. The readout gives the value for 1.5B, 7B, 14B and FinBERT, the share of "neutral" answers of the 1.5B model and the McNemar comparison of 7B with 1.5B.

        Question for the room Start at prompt P1 and run ▶ through P3. Among the Qwen2.5 models 1.5B, 7B and 14B, which will have the highest Twitter macro-F1 at P3?
        Answer

        14B. At P3, macro-F1 is 57.8% for 1.5B, 71.8% for 7B and 73.3% for 14B. At P1, 7B led: 72.6%, against 71.3% for 14B and 55.3% for 1.5B. The ranking also depends on the prompt: the advantage observed at P1 does not automatically carry over to P3.

        On PhraseBank, macro-F1 rises with size from 63.3% (0.5B) to 82.1% (14B), still below FinBERT's 88.3%, a lead that is partly training overlap (In-Sample Evaluation Bias). On Twitter the relation is not monotone with prompt P1: 1.5B is the worst model (55.3%) and 7B the best (72.6%), above FinBERT (66.8%) and above 14B (71.3%). In accuracy, 7B beats 1.5B by 21.5 points, with McNemar \(p < 0.001\).

        The ranking also depends on the prompt. At P3, macro-F1 is 57.8% for 1.5B, 71.8% for 7B and 73.3% for 14B, so the advantage of 7B at P1 does not carry over. Small models react strongly to wording: with prompt P1 the 1.5B model rarely answers "neutral". The prompt is a parameter of the classifier, discussed in Further Topics.

        In brief

        • Macro-F1 of zero-shot Qwen2.5 with the same prompt; dotted lines: FinBERT
        • PhraseBank: macro-F1 rises from 63.3% (0.5B) to 82.1% (14B), still below FinBERT (88.3%)
        • Twitter: not monotone
          • 1.5B is the worst (55.3%)
          • 7B the best (72.6%), above FinBERT (66.8%)
          • 7B against 1.5B on accuracy: +21.5 points, McNemar \(p\) \(<\)0.001
        • Small models react strongly to wording
          • The 1.5B model rarely answers “neutral” with this prompt
        MFM_ch15_llm_zero_shot

        Section summary

        Language models read words in context. FinBERT, a BERT fine-tuned on PhraseBank, looks better on PhraseBank than it is: its excess accuracy drop on Twitter, (15.6), is 9.2 points. Zero-shot LLMs classify from a prompt by reading the probabilities of the answer words, (15.7), without generating text. Their accuracy does not rise monotonically with size and depends on the prompt.

        Embeddings as Features

        An embedding model maps a text to a generic vector that is not trained for sentiment. A classifier fitted on these vectors with a modest number of labels is a fourth route between dictionaries and fine-tuned models. This section asks whether such vectors contain the sentiment of a headline and how many labels a simple classifier needs.

        Can Generic Text Vectors Measure Sentiment?

        Slide 23 in the PDF

        An embedding is a vector \(e_d \in \mathbb{R}^k\) assigned to text \(d\) such that similar texts have nearby vectors. Word-level embeddings go back to word2vec (Mikolov et al., 2013); Sentence-BERT (Reimers & Gurevych, 2019) produces vectors for whole sentences. The chapter uses MiniLM (Wang et al., 2020), with \(k = 384\) and vectors of unit length, so the similarity of two texts is \(\cos(e_a, e_b) = e_a^\top e_b\). Embeddings enter a classifier or a regression like any other feature.

        The chart shows the Twitter validation headlines in the plane of the first two principal components of their MiniLM vectors: the first component on the horizontal axis, the second on the vertical axis, each headline a point coloured by its human label (1,566 neutral, 475 positive and 347 negative). Principal component analysis (PCA), introduced in Chapter 3, keeps the directions of largest variance, not those tied to the label.

        In the interactive version, the slider rotates a projection direction within the PC1–PC2 (second Principal Component) plane, from 0° (the first principal component) to 180°, and the second panel shows the positive and negative headlines projected on that direction. The ▶ button rotates the direction step by step; the readout reports the area under the ROC curve (AUC) for positive against negative headlines at the current direction and at the best one.

          Question for the room The best direction in the plane of the first two principal components of the MiniLM vectors separates positive from negative headlines with AUC ≈ 0.636. What does this imply about the sentiment information in the embeddings?
          Answer

          Linear separation in this plane is modest, but the result does not rule out useful information in the other dimensions. PCA maximises explained variance without using the labels. Here the AUC measures only the ranking of positive against negative headlines; it is not three-class accuracy. A logistic regression on all 384 dimensions reaches 78.6% accuracy on Twitter.

          The best direction in this plane separates positive from negative headlines with AUC ≈ 0.636, a modest separation. This AUC concerns only the ranking of positive against negative headlines and is not a three-class accuracy. PCA maximises explained variance without using the labels, so the result does not rule out sentiment information in the other dimensions. A logistic regression on all 384 dimensions finds the sentiment direction and reaches 78.6% accuracy on Twitter.

          In brief

          • Embedding: a vector \(e_d \in \mathbb{R}^k\), close for similar texts
          • Chart: Twitter validation headlines, first two principal components of the MiniLM vectors
            • Coloured by label
          • The first two components do not separate sentiment
            • PCA (principal component analysis, Chapter 3) keeps the directions of largest variance
            • Not those tied to the label
          • A logistic regression on all 384 dimensions finds the sentiment direction: 78.6% accuracy on Twitter
          MFM_ch15_embeddings

          How Many Labels Do You Need? (1/2)

          Slide 24 in the PDF

          A classifier on embeddings or on TF-IDF features needs labels from the target domain, and the learning curve shows how its accuracy grows with their number. The horizontal axis is the number of labelled training headlines, on a log scale from 100 to the full Twitter training set; the vertical axis is the accuracy on the validation set in percent. The two coloured lines are MiniLM embeddings + logistic regression and TF-IDF + logistic regression, each with a band of \(\pm 2\) standard deviations across training resamples. The horizontal lines are FinBERT and Qwen2.5-7B, which use no Twitter labels.

          In the interactive version, the slider steps through the tested numbers of training labels, and the ▶ button moves along the curve. The readout reports the mean accuracy of both classifiers at the chosen size and compares them with FinBERT.

            Question for the room What is the first tested number of Twitter labels at which MiniLM + LR's mean accuracy across training resamples exceeds FinBERT?
            Answer

            400. The mean is 71.9% at 200 labels and 73.3% at 400; FinBERT has 72.5%. Some training runs already beat the benchmark at 200, and at 800 all ten do. The threshold of 400 concerns the mean, not every sample. TF-IDF starts lower (66.9% at 100 labels) but reaches about 83% with all 9,543 labels, against 78.6% for embeddings.

            The mean accuracy of MiniLM + LR is 71.9% at 200 labels and 73.3% at 400, against 72.5% for FinBERT: 400 is the first tested size at which the mean exceeds the benchmark. Some training runs already beat FinBERT at 200 labels, and at 800 all ten do. The threshold of 400 concerns the mean, not every sample.

            How Many Labels Do You Need? (2/2)

            Slide 25 in the PDF

            Each point of the curve comes from a logistic regression trained on a random subset of the Twitter training set of the given size, with 10 repetitions per size; the band covers \(\pm 2\) standard deviations across the repetitions. The design measures how much the result depends on which headlines happen to be labelled, as well as on how many.

            Embeddings + logistic regression start high: 69.0% with 100 labels, and above FinBERT from 400 labels. The pretrained vectors already contain most of what the classifier needs, and a few hundred labels suffice to find the sentiment direction. TF-IDF starts lower, at 66.9%, but keeps improving and reaches about 83% with all labels, against 78.6% for embeddings. With many labels, exact words such as "upgrade", "cuts" or "misses" carry the signal in headlines, and a sparse word representation can use them directly.

            The practical rule is to label a few hundred documents from one's own domain before trusting any off-the-shelf model. The same labelled sample serves later as the validation sample of the section on scores as estimates.

            In brief

            • Logistic regression trained on random subsets of the Twitter training set
              • 10 repetitions per size; band: \(\pm 2\) standard deviations
            • Embeddings + logistic regression: 69.0% with 100 labels, above FinBERT from 400 labels
              • The pretrained vectors already contain most of what is needed
            • TF-IDF starts lower (66.9%) but keeps improving
              • About 83% with all labels, against 78.6% for embeddings
              • With many labels, exact words (“upgrade”, “cuts”, “misses”) carry the signal in headlines
            • Practical rule: label a few hundred documents from your own domain
              • Do this before trusting any off-the-shelf model
            MFM_ch15_embeddings

            Section summary

            Generic MiniLM vectors do not separate sentiment in their first two principal components, but a logistic regression on all 384 dimensions reaches 78.6% on Twitter and exceeds FinBERT in mean accuracy from 400 labels. With the full training set, TF-IDF does better still, at about 83%. A few hundred labels from the domain are worth more than a larger general model.

            Look-Ahead Bias and Memorisation

            An LLM is trained on text that may describe the events a backtest tries to forecast. This section states the problem, presents a direct test of what Qwen2.5 remembers about past market moves, and lists the remedies.

            Memorisation of Test Data by Language Models

            Slide 26 in the PDF

            An LLM trained on text up to 2024 may have read what happened after a headline published in 2015. A backtest on 2015 news then mixes forecasting with remembering: a high score may reflect knowledge of the subsequent price path rather than a reading of the headline.

            Glasserman & Lin (2023) identify two biases in sentiment backtests with LLMs. Look-ahead bias arises when the model knows the returns that followed the news. Distraction arises when general knowledge about the named company interferes with reading the text; in their tests, distraction mattered more than look-ahead. Sarkar & Vafa (2024) show, with direct tests, that look-ahead is present in LLM applications to earnings calls and elections.

            Every backtest of LLM scores on headlines that predate the model's release therefore needs a statement of what the model could have known, and preferably a test of it.

            In brief

            • An LLM trained on text up to 2024 may have read what happened after a 2015 headline
              • A backtest on 2015 news then mixes forecasting with remembering
            • Glasserman & Lin (2023): two biases in sentiment backtests
              • Look-ahead: the model knows the returns that followed the news
              • Distraction: general knowledge about the named company interferes with reading the text; in their tests it mattered more than look-ahead
            • Sarkar & Vafa (2024): direct tests show look-ahead in LLM applications to earnings calls and elections

            Our Test: Ask About the Past, Without Context

            Slide 27 in the PDF

            The test asks the model about past market moves without giving it any text to read. In the monthly version the question is "Did the S&P 500 index rise or fall during <month year>?", asked for every month in 296 months before the release of the weights and 23 months after it. In the daily version the question is "Did Apple stock close higher or lower on <date> than on the previous trading day?", asked for 500 random days from 2010–2023 and 493 days after October 2024.

            The score is \(P(\text{rise})\), computed from the probabilities of the answer words as in (15.7). The true direction comes from the monthly closes of the S&P 500 and the daily closes of Apple. Discrimination is measured by the AUC, the area under the ROC (receiver operating characteristic) curve: the probability that a randomly chosen up period receives a higher score than a randomly chosen down period. An AUC of 0.5 means no discrimination; the metric is presented in Chapter 13.

            After the release there is nothing to remember. If the date carries no predictive information, for example through seasonality, the population AUC after the release is 0.5 and the sample AUC varies around it. The months after the release thus act as a control for the months before.

            In brief

            • Monthly: “Did the S&P 500 index rise or fall during <month year>?”
              • Every month from 296 pre-release months plus 23 after
              • Daily: “Did Apple stock close higher or lower on <date> than on the previous trading day?”, 500 random days 2010–2023 and 493 days after October 2024
            • Score: \(P(\text{rise})\) from the answer-word probabilities
              • True direction from the S&P 500 monthly closes and the Apple daily closes
              • AUC (area under the ROC curve; ROC: receiver operating characteristic): probability that a random up period gets a higher score than a random down period; 0.5 = no discrimination (Chapter 13)
            • After the release there is nothing to remember
              • If the date carries no predictive information (e.g. seasonality), the population AUC is 0.5
              • The sample AUC varies around it

            Memory Measured: Months Yes, Days No (1/2)

            Slide 28 in the PDF

            The chart reports the AUC of three Qwen2.5 models, 1.5B, 7B and 14B, with 95% intervals. The left panel concerns the direction of S&P 500 months and the right panel the direction of single Apple days; the horizontal axis is the model size and the vertical axis the AUC. Blue points are periods before the weights were released, orange points periods after the release, when the model cannot know the answer. The dashed line marks 0.5.

            In the interactive version, the first selector chooses the series (S&P 500 months or Apple days) and the second the model. The slider sets the threshold above which an answer counts as "rise", and the second panel shows the corresponding point on the ROC curve; the ▶ button moves the threshold from 0 to 1 and traces the curve.

              Question for the room For Qwen2.5-14B, the AUC on the S&P 500 months before the weights were released is 0.69, with a 95% CI of [0.62; 0.75]. What conclusion does this result support?
              Answer

              Discrimination above 0.5, consistent with memorisation, without identifying the mechanism on its own. The result measures how well past months are discriminated: it does not separate memorisation from calendar or regime regularities, nor does it establish future performance. For single days of Apple, the AUC stays between 0.48 and 0.52. On the 23 months after the release, the AUC is 0.59, with a wide interval, approximately [0.35; 0.84]: about 101 months would be needed.

              For Qwen2.5-14B, the AUC on the S&P 500 months before the release is 0.69, with 95% CI [0.62; 0.75]. This is discrimination above 0.5, consistent with memorisation, but it does not identify the mechanism on its own: it does not separate memorisation from calendar or regime regularities, and it says nothing about future months. For single Apple days the AUC stays between 0.48 and 0.52. On the 23 months after the release, the AUC of 14B is 0.59, with a wide interval of about [0.35; 0.84].

              Memory Measured: Months Yes, Days No (2/2)

              Slide 29 in the PDF

              The intervals are bootstrap 95% intervals, with months resampled independently. Before the release, the AUC is 0.67 for 7B (CI [0.61; 0.73]) and 0.69 for 14B (CI [0.62; 0.75]), and the permutation p-value for 14B is below 0.001. The 1.5B model knows much less, with an AUC of 0.57.

              After the release, on 23 months, the intervals include 0.5, but they are too wide to prove anything. About 101 post-release months would be needed to detect a fall from 0.69 to 0.5; the formal test and its power are in the Appendix. For single Apple days, the AUC lies between 0.48 and 0.52 before the release: no discrimination above chance is detected.

              Look-ahead is thus a danger at the horizon of well-known events, such as the direction of a month remembered for a crash, and less so for daily noise. The result is descriptive: the discrimination of months is compatible with memorisation but does not separate it from calendar or regime regularities. The month-by-month answers of 14B are shown in Further Topics.

              In brief

              • AUC with bootstrap 95% intervals, months resampled independently
              • Months before release: AUC 0.67 (7B, CI [0.61; 0.73]) and 0.69 (14B, CI [0.62; 0.75])
                • Permutation \(p\) for 14B \(<\)0.001
                • The 1.5B model knows much less (0.57)
              • After release (23 months) the intervals include 0.5 but are too wide to prove anything
                • About 101 post-release months would be needed to detect a fall to 0.5 (formal test: Appendix)
              • Single days of Apple: AUC between 0.48 and 0.52 before release, no discrimination above chance detected
                • Look-ahead is a danger at the horizon of well-known events, less for daily noise
              • Type of evidence: description; the discrimination of months is compatible with memorisation but does not separate it from calendar or regime regularities
              MFM_ch15_lookahead

              Remedies, Including Point-in-Time Models

              Slide 30 in the PDF

              Three remedies are available:

              1. Testing only on news published after the model's training cutoff, the cleanest remedy. Lopez-Lira & Tang (2026) evaluate GPT-4 on headlines published after its cutoff; the design is clean, but the evaluation window starts late.
              2. Masking company names and dates before scoring and comparing slopes with and without masking (Glasserman & Lin, 2023). This is a diagnostic rather than a cure, since products, events or quotes can still identify the episode.
              3. A point-in-time model, trained only on text available up to the date \(t\) of the analysis (Sarkar & Vafa, 2024). Drinkall et al. (2024) release a series of point-in-time LLMs, each uninformed about facts after its date, with public models and training data. He et al. (2025) build ChronoBERT and ChronoGPT, which use only text available at each date; on next-day returns from news, their Sharpe ratios are close to those of a much larger model, so look-ahead is modest in that application.

              Whatever remedy is chosen, every result should state the model version and the release date of its weights, so that the reader can judge what the model could have known.

              In brief

              • Test only after the model's training cutoff
                • Lopez-Lira & Tang (2026) evaluate GPT-4 (Generative Pre-trained Transformer 4) on post-cutoff headlines
                • Clean, but the window starts late
              • Mask company names and dates before scoring (Glasserman & Lin (2023))
                • Then compare slopes with and without masking
                • A diagnostic, since products, events or quotes can still identify the episode
              • Point-in-time models, trained only on text available up to date \(t\) (Sarkar & Vafa (2024))
                • Drinkall et al. (2024): a series of point-in-time LLMs, each uninformed about facts after its date; models and training data are public
                • He et al. (2025): ChronoBERT and ChronoGPT use only text available at each date; on next-day returns from news their Sharpe ratios are close to a much larger model: look-ahead there is modest
              • Report the model version and the release date of its weights with every result

              Section summary

              Asked only for the month, Qwen2.5-14B separates up from down months of the S&P 500 before its release with an AUC of 0.69, while single Apple days are not discriminated. The 23 post-release months are too few to show whether the discrimination vanishes. Backtests of LLM scores therefore need post-release data, masking diagnostics or point-in-time models, and a statement of the model's release date.

              From Sentiment to a Trading Signal

              The previous sections measured how well scores agree with human labels. This section asks whether the tone of dated headlines predicts returns that can still be traded. It describes the FNSPID headlines and their timestamps, the pooled regression and its inference with 16 stocks, and a daily long-short strategy after costs.

              Dated Headlines Matched with Daily Prices

              Slide 31 in the PDF

              The data come from FNSPID, the Financial News and Stock Price Integration Dataset (Dong, Fan & Peng, 2024): 15.7 million news records about S&P 500 firms from four news websites, covering 1999–2023, under the Creative Commons licence CC BY-NC 4.0 (attribution, non-commercial use).

              The chapter uses 16 large US stocks for which daily prices are available in the course data: technology firms, banks, GameStop and Strategy. After repeated headlines are removed, there are 106,820 distinct headlines and 18,282 stock-days with news, from 27 October 2009 to 18 December 2023. Coverage is uneven over time, as Further Topics shows.

              The sample is small in the cross-section and selected: 16 heavily followed large firms, chosen among today's survivors. Both features matter for the interpretation of the results that follow.

              In brief

              • The question of this section: does headline tone predict returns that we can still trade?
              • FNSPID (Financial News and Stock Price Integration Dataset, Dong, Fan & Peng (2024))
                • 15.7 million news records about S&P 500 firms, 1999–2023, from four news websites
                • Creative Commons licence CC BY-NC 4.0 (attribution, non-commercial use)
              • Here: 16 large US stocks with daily prices in the course data
                • Technology, banks, GameStop, Strategy
                • 106 820 distinct headlines after removing repeats; 18 282 stock-days with news, 27.10.2009 to 18.12.2023
              Dated Headlines Matched with Daily Prices News ticker on the Reuters building, New York Photo: bgilliard (2005); CC BY-SA 2.0; Wikimedia Commons

              News Timestamps: Dates without a Time of Day (1/2)

              Slide 32 in the PDF

              The timing of a headline relative to the trading session decides which return it can predict. The chart shows the distribution of the FNSPID timestamps by hour. The horizontal axis is the hour of the timestamp in Coordinated Universal Time (UTC), the vertical axis the number of headlines on a log scale; the red bar is midnight, 00:00 UTC.

              In the interactive version, the slider moves the clock from UTC to another time zone, with offsets from −12 to +12 hours, and the ▶ button steps through the offsets.

                Question for the room If we move the clock from UTC to UTC−5, do we learn whether the FNSPID headlines appeared before or after the close?
                Answer

                No. 99.8% of the headlines are stamped exactly 00:00 UTC, so the source gives only the date. At UTC−5, 00:00 UTC becomes 19:00 on the previous day, but the conversion only changes how the stamp is displayed, without adding the publication time. The returns of days d and d+1 may therefore still contain the reaction, and the first return used as a tradable forecast is that of day d+2.

                Moving the clock does not reveal whether a headline appeared before or after the close. 99.8% of the headlines are stamped exactly 00:00 UTC, so the source gives only the date. At UTC−5, 00:00 UTC becomes 19:00 on the previous day, but the conversion only changes how the stamp is displayed; it adds no publication time.

                News Timestamps: Dates without a Time of Day (2/2)

                Slide 33 in the PDF

                Because 99.8% of headlines carry the stamp 00:00 UTC, the source records the date of publication, not its time. A headline dated \(d\) may have appeared before the open, during the session or after the close of day \(d\). Any alignment of headlines with returns has to be valid in all three cases.

                The alignment used in the chapter assigns each headline to day \(d\), the first trading day on or after its date. Returns on \(d\) and on \(d+1\) may still contain the reaction to the news: on \(d\) if the headline appeared before or during the session, on \(d+1\) if it appeared after the close. The first safe forecast is the return on \(d+2\), from the close of \(d+1\) to the close of \(d+2\). This alignment has no look-ahead, at the price of discarding the first day after the news.

                In brief

                • Hour of the FNSPID timestamp in Coordinated Universal Time (UTC), log scale
                • 99.8% of headlines are stamped exactly 00:00 UTC
                  • The source gives the date, not the time of publication
                  • A headline dated \(d\) may have appeared before the open, during the session or after the close of day \(d\)
                • Alignment used here, with no look-ahead
                  • Day \(d\): the first trading day on or after the date of the headline
                  • Returns on \(d\) and \(d+1\) may still contain the reaction
                  • First safe forecast: the return on \(d+2\), close of \(d+1\) to close of \(d+2\)
                MFM_ch15_news_data

                Three Scores and the Excess Return

                Slide 34 in the PDF

                Three scores are computed for every headline: the LM tone (15.2), the FinBERT net score \(P(\text{pos}) - P(\text{neg})\), and the Qwen2.5-7B net score with prompt P1. The LM dictionary finds a listed word in only 27% of the headlines; all the others receive tone 0. The FinBERT labels are 19% negative, 62% neutral and 18% positive.

                The scores agree only partly. Across headlines, the correlation is 0.62 between FinBERT and Qwen, 0.41 between FinBERT and LM and 0.36 between Qwen and LM. The two language models are closer to each other than either is to the dictionary.

                The outcome is the excess return of stock \(i\) on day \(t\) over SPY, an exchange-traded fund that tracks the S&P 500 index:

                \[r^{e}_{i,t} = r_{i,t} - r_{\text{SPY},t}.\]
                (15.9)

                This market-adjusted return sets \(\beta = 1\) for every stock. The usual alternative is \(r_{i,t} - \hat\alpha_i - \hat\beta_i r_{\text{SPY},t}\), with \(\hat\beta_i\) estimated before the event.

                In brief

                • The three scores: LM tone, FinBERT net score and Qwen2.5-7B net score with prompt P1 (the one on the zero-shot classification slide)
                  • FinBERT net score: \(P(\text{pos}) - P(\text{neg})\)
                  • LM finds a listed word in only 27% of headlines: the rest get tone 0
                  • FinBERT labels: 19% negative, 62% neutral, 18% positive
                  • Correlations across headlines: FinBERT–Qwen 0.62, FinBERT–LM 0.41, Qwen–LM 0.36
                • Excess return: \(r^{e}_{i,t} = r_{i,t} - r_{\text{SPY},t}\)
                  • SPY: an exchange-traded fund tracking the S&P 500 index
                  • Market-adjusted: \(\beta = 1\) for every stock; the usual alternative is \(r_{i,t} - \hat\alpha_i - \hat\beta_i r_{\text{SPY},t}\), with \(\hat\beta_i\) estimated before the event

                The Regression and Standard Errors Clustered by Day

                Slide 35 in the PDF

                The relation between scores and returns is estimated by a pooled regression over the stock-days with news:

                \[r^{e}_{i,d+h} = a + b\,z_{i,d} + \varepsilon_{i,d+h}, \qquad h = 0, 1, 2,\]
                (15.10)

                where \(z_{i,d}\) is the daily score of stock \(i\) on day \(d\), standardised over the panel to mean 0 and standard deviation 1. The slope \(b\) is then the excess return, in percent, per one standard deviation of the score. The horizon \(h = 0\) measures the reaction on the news day, and \(h = 2\) measures predictability, since \(d+2\) is the first safe return.

                The errors of different stocks on the same day are correlated through common market shocks. Ordinary least squares (OLS) standard errors ignore this correlation and are too small. Following Petersen (2009), the variance is clustered by day:

                \[\widehat{V} = (X^\top X)^{-1}\Big(\sum_{g} X_g^\top \hat\varepsilon_g \hat\varepsilon_g^\top X_g\Big)(X^\top X)^{-1},\]
                (15.11)

                where \(g\) indexes trading days, \(X\) contains the regressors (a constant and \(z\)), and \(X_g\) and \(\hat\varepsilon_g\) are the rows and OLS residuals of day \(g\). The middle term allows any correlation among the residuals of one day and none across days.

                In brief

                • Pooled regression over stock-days with news: \(r^{e}_{i,d+h} = a + b\,z_{i,d} + \varepsilon_{i,d+h}\), \(h = 0, 1, 2\)
                  • \(z\): daily score standardised over the panel (mean 0, standard deviation 1), so \(b\) is the excess return (%) per one standard deviation of score
                  • \(h = 0\) measures reaction, \(h = 2\) measures predictability
                • Errors of stocks on the same day are correlated (common market shocks)
                  • OLS (ordinary least squares) standard errors (SEs) ignore this: cluster by day (Petersen (2009))
                  • \(\widehat{V} = (X^\top X)^{-1}\left(\sum_{g} X_g^\top \hat\varepsilon_g \hat\varepsilon_g^\top X_g\right)(X^\top X)^{-1}\)
                  • \(g\): trading days; \(X\): the regressors (constant, \(z\)); \(X_g\), \(\hat\varepsilon_g\): the rows and OLS residuals of day \(g\)

                Reaction versus Prediction: Headlines Describe Prices

                Slide 36 in the PDF

                The chart reports the slope \(b\) of (15.10) for each of the three scores and three horizons. The horizontal axis lists the scores (Loughran–McDonald tone, FinBERT, Qwen2.5-7B); the vertical axis is the excess return in basis points (bp, 1 bp = 0.01%) per one standard deviation of the score, with 95% intervals clustered by day. Blue is the news day \(d\), ochre the day \(d+1\), when news may still be arriving, and red the day \(d+2\), the tradable forecast.

                In the interactive version, the slider sets the minimum number of headlines per stock-day that enters the regression, from 1 to 20, and the selector chooses the horizons, \(d\) to \(d+2\) or \(d\) to \(d+5\). The ▶ button raises the minimum number of headlines step by step.

                  Question for the room With the initial filter of at least one headline per stock-day, what conclusion do the slope estimates for the excess return of day d+2 support?
                  Answer

                  We do not detect slopes different from zero at the two-sided 5% level, with standard errors clustered by day. The t-statistics are −0.25 for FinBERT, +1.19 for Qwen and −0.01 for LM; none exceeds the asymptotic reference value 1.96 in absolute value. This is a lack of evidence in this specification, not proof of exactly zero effects. On day d the slopes are large: FinBERT +39.2 bp (t = 17.27) and Qwen +41.4 bp (t = 19.07); many headlines report the price move itself.

                  On day \(d\) the slopes are large: FinBERT +39.2 bp (\(t\) = +17.27), Qwen +41.4 bp (\(t\) = +19.07) and LM +15.5 bp (\(t\) = +8.53). Many headlines report the price move itself ("shares jump", "stock falls"), so causality runs both ways. On day \(d+1\) the slopes are +3.2 bp for FinBERT (\(t\) = +1.60) and +3.7 bp for Qwen (\(t\) = +1.86); part of this is news published after the close of \(d\), not a forecast.

                  On day \(d+2\) the slopes are −0.5 bp for FinBERT (\(t\) = −0.25), +2.3 bp for Qwen (\(t\) = +1.19) and −0.0 bp for LM (\(t\) = −0.01). None exceeds 1.96 in absolute value: we do not detect slopes different from zero at the two-sided 5% level, with standard errors clustered by day. This is a lack of evidence in this specification, not proof of exactly zero effects.

                  In brief

                  • Slope \(b\) in basis points (bp, 1 bp = 0.01%) per one standard deviation of score (s.d. in the chart)
                    • 95% CI clustered by day
                  • Day \(d\): FinBERT +39.2 bp (\(t\) = +17.27), Qwen +41.4 bp (\(t\) = +19.07), LM +15.5 bp (\(t\) = +8.53)
                    • Many headlines report the price move itself (“shares jump”, “stock falls”): causality runs both ways
                  • Day \(d+1\): FinBERT +3.2 bp (\(t\) = +1.60), Qwen +3.7 bp (\(t\) = +1.86)
                    • Part of this is news published after the close of \(d\), not a forecast
                  • Day \(d+2\): FinBERT \(-\)0.5 bp (\(t\) = \(-\)0.25), Qwen +2.3 bp (\(t\) = +1.19), LM \(-\)0.0 bp (\(t\) = \(-\)0.01)
                  MFM_ch15_news_signal

                  Inference with 16 Stocks: Clustering by Stock and Two-Way

                  Slide 37 in the PDF

                  Clustering by day handles common shocks, but the errors of one stock may also be correlated across days, because firm effects persist. The slopes therefore have to survive clustering by stock as well. Two-way clustering (Cameron, Gelbach & Miller, 2011) combines the two:

                  \[\widehat V_{\text{2way}} = \widehat V_{\text{day}} + \widehat V_{\text{stock}} - \widehat V_{\text{day} \cap \text{stock}},\]
                  (15.12)

                  where each term is the sandwich (15.11) with clusters given by day, by stock and by the stock–day pair respectively.

                  A stock–day pair contains a single observation here, so the last term is the White estimator, robust only to heteroskedasticity. Subtracting it avoids counting the variance of each observation twice, since both the day and the stock clusters include it. Driscoll & Kraay (1998) also allow for correlation across different days.

                  In brief

                  • The question: do the slopes survive if errors are also correlated over time within a stock?
                  • Firm effects persist, so the errors of one stock are correlated across days: cluster by stock as well
                  • Two-way clustering (Cameron, Gelbach & Miller (2011)): \(\widehat V_{\text{2way}} = \widehat V_{\text{day}} + \widehat V_{\text{stock}} - \widehat V_{\text{day} \cap \text{stock}}\)
                    • Each term is the sandwich formula of the previous slide, with clusters given by day, by stock, and by the stock–day pair
                    • A stock–day pair holds a single observation here, so the last term is the White estimator, robust to heteroskedasticity only
                    • Subtracting it avoids counting the variance of each observation twice
                  • Driscoll & Kraay (1998) also allow for correlation across different days

                  Few Clusters: the Wild Cluster Bootstrap

                  Slide 38 in the PDF

                  Clustering by stock leaves only \(G = 16\) clusters. With so few clusters, the stock-clustered standard error is too small and the \(t\) test rejects too often (Cameron, Gelbach & Miller, 2008). The wild cluster bootstrap with the null imposed corrects this.

                  The procedure first fits the restricted model with \(b = 0\), which gives a constant \(\tilde a\) and residuals \(\tilde\varepsilon_{i,t}\). For \(j = 1, \dots, B\), with \(B = 9{,}999\), it draws Rademacher weights \(w_g = \pm 1\) with probability \(\tfrac12\) each, one draw per stock \(g\), builds the bootstrap returns \(r^{e*}_{i,t} = \tilde a + w_{g(i)}\,\tilde\varepsilon_{i,t}\), re-fits the regression and computes the stock-clustered statistic \(t^*_j\). The p-value is

                  \[p = \frac{1}{B}\sum_{j=1}^{B} \mathbf{1}\{|t^*_j| \ge |t|\}.\]
                  (15.13)

                  Giving all residuals of a stock the same sign keeps the correlation within the cluster. The bootstrap assumes that stocks are independent, so it is read next to the two-way statistic, which allows correlation across stocks on the same day.

                  In brief

                  • With only \(G = 16\) stocks, the stock-clustered SE (Standard Error) is too small and the \(t\) test rejects too often (Cameron, Gelbach & Miller (2008))
                  • Wild cluster bootstrap (resampling with random signs per cluster) with the null imposed: fit the restricted model (\(b = 0\))
                    • Constant \(\tilde a\) and residuals \(\tilde\varepsilon_{i,t}\)
                    • For \(j = 1, \dots, B\), \(B = 9{,}999\): Rademacher weights \(w_g = \pm 1\) with probability \(\frac12\), one draw per stock \(g\)
                    • \(r^{e*}_{i,t} = \tilde a + w_{g(i)}\,\tilde\varepsilon_{i,t}\); re-fit the regression and compute the stock-clustered \(t^*_j\)
                    • \(p = \frac{1}{B}\sum_j \mathbf{1}\{|t^*_j| \ge |t|\}\)
                  • The same sign for all residuals of a stock keeps the correlation within the cluster
                  • The bootstrap assumes independent stocks, so read it next to the two-way \(t\)

                  Nine Regressions: the Holm Correction

                  Slide 39 in the PDF

                  The analysis runs nine regressions: three scores times three horizons, \(h = 0, 1, 2\). If no score had any effect, nine tests at 5% would still give \(9 \times 0.05 = 0.45\) false rejections on average. The family of tests needs a correction for multiple testing.

                  The Holm correction, presented in Chapter 2, keeps the family-wise error rate (FWER), the probability of any false rejection, at or below 5%. With the p-values sorted, \(p_{(1)} \le \dots \le p_{(m)}\) and \(m = 9\), the adjusted p-values are

                  \[\tilde p_{(k)} = \max_{j \le k}\ \min\{1,\ (m - j + 1)\,p_{(j)}\},\]
                  (15.14)

                  and \(H_{(k)}\) is rejected when \(\tilde p_{(k)} \le 0.05\). Holm is more powerful than Bonferroni, which multiplies every p-value by \(m\), and remains valid under any dependence between the tests.

                  In brief

                  • Nine regressions: 3 scores \(\times\) 3 horizons (\(h\) = 0, 1, 2)
                  • If no score had any effect, nine tests at 5% would still give \(9 \times 0.05 = 0.45\) false rejections on average
                  • The Holm correction (Chapter 2) keeps the FWER at or below 5%
                    • FWER (family-wise error rate): the chance of any false rejection
                    • Sort the \(p\)-values: \(p_{(1)} \le \dots \le p_{(m)}\), with \(m = 9\)
                    • \(\tilde p_{(k)} = \max_{j \le k} \min\{1, (m - j + 1)\,p_{(j)}\}\); reject \(H_{(k)}\) when \(\tilde p_{(k)} \le 0.05\)
                    • More powerful than Bonferroni (\(m\,p_{(k)}\) for all) and valid under any dependence between the tests

                  Inference with 16 Stocks and Nine Regressions: Results

                  Slide 40 in the PDF

                  The table reports, for the main combinations of score and horizon, the slope in basis points, the \(t\) statistic clustered by stock, the two-way statistic (15.12), the wild-bootstrap p-value (15.13) and the Holm-adjusted p-value (15.14). The same-day slopes survive every correction: FinBERT +39.2 bp, with \(t\) = +5.71 by stock and +5.67 two-way, and Qwen +41.4 bp, with +5.10 and +5.07; their bootstrap and Holm p-values are below 0.001.

                  The tradable horizon is different. Qwen at \(d+2\) has \(t\) = +2.41 by stock, which looks significant: \(p\) = 0.029 under a \(t\) distribution with 15 degrees of freedom. The wild bootstrap gives \(p\) = 0.061, and Holm then gives 0.37. Among the nine sorted bootstrap p-values, Qwen at \(d+2\) comes fourth, after the three same-day values, all below 0.001, so its Holm p-value is \((9 - 4 + 1) \times 0.061 \approx 0.37\). FinBERT and LM at \(d+2\) have Holm p-values of 1.00. The rows not shown are LM at \(d\), with \(p < 0.001\), and the three slopes at \(d+1\), with \(p\) = 0.272 (FinBERT), 0.317 (Qwen) and 0.802 (LM).

                  The result supports association: scores and same-day returns move together, but the direction of causality is open. The event study in Further Topics shows that the stock-days with the most positive scores already gain +0.63% in the 5 days before the news.

                  In brief

                  Score, horizon\(b\) (bp)\(t\) by stock\(t\) two-way\(p\) wild bootstrap\(p\) Holm
                  FinBERT, \(d\)+39.2+5.71+5.67\(<\)0.001\(<\)0.001
                  Qwen, \(d\)+41.4+5.10+5.07\(<\)0.001\(<\)0.001
                  FinBERT, \(d+2\)\(-\)0.5\(-\)0.35\(-\)0.300.7391.00
                  Qwen, \(d+2\)+2.3+2.41+2.010.0610.37
                  LM, \(d+2\)\(-\)0.0\(-\)0.03\(-\)0.020.9791.00
                  • Qwen at \(d+2\): \(t\) = +2.41 by stock looks significant
                    • \(p\) = 0.029 under a \(t\) distribution with 15 degrees of freedom
                    • But not after the wild bootstrap and Holm
                    • Holm step by step: sort the 9 bootstrap \(p\)-values; Qwen at \(d+2\) comes 4th, after the three same-day values (all below 0.001), so its Holm \(p\) = (9 \(-\) 4 + 1) \(\times\) 0.061 \(\approx\) 0.37
                    • Rows not shown: LM at \(d\), \(p\) \(<\)0.001; at \(d+1\), \(p\) = 0.272 (FinBERT), 0.317 (Qwen), 0.802 (LM)
                  • Type of evidence: association (the variables move together; this does not prove causation)
                    • The most positive third of scores already has a cumulative excess return of +0.63% in the 5 days before the news (Further Topics)

                  A Daily Long-Short Sentiment Strategy

                  Slide 41 in the PDF

                  The regression measures association; a strategy asks whether the sign of the score earns a profit that can actually be collected. At the close of \(d+1\), for every stock with news and a non-zero score on day \(d\), the strategy goes long if the score is positive and short if it is negative. Each position is hedged with SPY; a one-for-one hedge removes the market move only when \(\beta = 1\). Positions have equal weights and are closed after one day, at the close of \(d+2\). The LM score is zero on 48% of stock-days, and these stock-days do not enter the portfolio.

                  The gross return of the strategy on day \(d+2\) is

                  \[G_{d+2} = \frac{1}{N_d}\sum_{i \in A_d} \operatorname{sign}(s_{i,d})\,\big(r_{i,d+2} - r_{\text{SPY},d+2}\big),\]
                  (15.15)

                  where \(A_d = \{i : |s_{i,d}| > b\}\) is the set of stocks with news and a score above the threshold \(b\) in absolute value, here \(b = 0\), and \(N_d = |A_d|\) is the number of positions opened at the close of \(d+1\). If \(A_d\) is empty, \(G_{d+2} = 0\) and no costs are paid.

                  In brief

                  • The question: does the sign of the score earn a profit we can actually collect?
                  • At the close of \(d+1\): for every stock with news and a non-zero score on \(d\), go long or short
                    • Long if its score is positive and short if negative
                    • Each position hedged with SPY; a one-for-one hedge removes the market move only when \(\beta = 1\)
                    • Equal weights; positions are closed after one day, at the close of \(d+2\)
                    • The LM score is zero on 48% of stock-days: these do not enter the portfolio
                  • Gross return on \(d+2\): \(G_{d+2} = \frac{1}{N_d}\sum_{i \in A_d} \operatorname{sign}(s_{i,d})\,(r_{i,d+2} - r_{\text{SPY},d+2})\)
                    • \(A_d = \{i : |s_{i,d}| > \delta\}\), with threshold \(\delta = 0\): the stocks with news and a non-zero score on day \(d\)
                    • \(N_d = |A_d|\): the positions opened at the close of \(d+1\); if \(A_d\) is empty, \(G_{d+2} = 0\) and no costs are paid

                  Trading Costs: Round Trips and the Break-Even Cost

                  Slide 42 in the PDF

                  The cost of trading is set at \(c\) = 5 bp per one-way trade, per unit of traded notional. The value is assumed, not estimated; it is plausible for large, liquid US stocks. A round trip, entering and leaving a position, costs \(2c\). On a day with positions the strategy makes one round trip for the stock book and one for the SPY hedge, each of notional 1, so it pays \(4c\) = 20 bp.

                  The net return is

                  \[G^{\text{net}}_{d+2} = G_{d+2} - 4c\,\mathbf{1}\{N_d > 0\}.\]
                  (15.16)

                  The break-even cost \(c^*\) is the value of \(c\) at which the mean net return is zero, \(c^* = \bar G_{\text{active}}/4\), where \(\bar G_{\text{active}}\) is the mean gross return on the days with positions. A strategy is worth trading only if its break-even cost exceeds the cost the trader actually pays.

                  In brief

                  • \(c\) = 5 bp per one-way trade, per unit of traded notional
                    • Assumed, not estimated; plausible for large, liquid US stocks
                  • A round trip (in and out) costs \(2c\)
                  • On a day with positions: one round trip for the stock book and one for the SPY hedge
                    • Each of notional 1, so \(4c\) = 20 bp
                  • Net return: \(G^{\text{net}}_{d+2} = G_{d+2} - 4c\,\mathbf{1}\{N_d > 0\}\)
                  • Break-even cost \(c^*\): the value of \(c\) at which the mean net return is 0
                    • I.e. \(c^* = \bar G_{\text{active}}/4\)
                    • \(\bar G_{\text{active}}\): the mean gross return on days with positions

                  Reading the Strategy Numbers: Sharpe Ratio and Summed Returns

                  Slide 43 in the PDF

                  Inference on the mean daily return uses the Newey–West \(t\) statistic, presented in Chapter 2, and a block-bootstrap confidence interval. Both allow for serial correlation of the daily returns.

                  The annualised Sharpe ratio is

                  \[SR = \sqrt{252}\,\frac{\bar G}{\hat\sigma},\]
                  (15.17)

                  where \(\bar G\) and \(\hat\sigma\) are the mean and standard deviation of the daily return \(G_t\) over all 3,568 trading days; days without positions enter as 0, and after costs \(G_t\) is replaced by the net return (15.16). For Qwen2.5-7B, a mean of +5.64 bp and \(SR\) = +0.63 imply \(\hat\sigma = \sqrt{252} \times \text{mean}/SR \approx 143\) bp per day.

                  The effect of costs follows from (15.16): with positions on 95% of days, the mean falls by \(20 \times 0.95\) bp. The next chart sums the daily simple excess returns; the sum is an additive measure of profit, not compounded wealth.

                  In brief

                  • Inference: Newey–West \(t\) (Chapter 2) and block-bootstrap CI for the mean daily return
                  • Sharpe ratio \(SR = \sqrt{252}\,\bar G / \hat\sigma\) (annualised)
                    • \(\bar G\), \(\hat\sigma\) are the mean and standard deviation of the daily return \(G_t\) over all 3 568 trading days
                    • Days without positions enter as 0
                    • After costs, \(G_t\) is replaced by the net return
                    • Example (Qwen2.5-7B): mean +5.64 bp and \(SR\) = +0.63 imply \(\hat\sigma = \sqrt{252} \times\) mean\(/SR \approx\) 143 bp per day
                    • After costs: positions on 95% of days, so the mean falls by \(20 \times\) 0.95 bp, to \(-\)13.36 bp per day
                  • The next chart sums daily simple excess returns
                    • An additive profit measure, not compounded wealth

                  Cumulative Strategy Returns: No Net Profit

                  Slide 44 in the PDF

                  The chart shows the cumulative sum of the daily excess returns of the strategy (15.15), in percent, from 2010 to 2024. Solid lines are gross returns and dashed lines net returns with 5 bp per one-way trade; orange is the LM score, blue FinBERT and red Qwen2.5-7B.

                  In the interactive version, the first slider chooses the return day, from \(d+1\) to \(d+5\), the second the cost per one-way trade, from 0 to 10 bp, and the third the last day included in the sum. The ▶ button extends the sum through time.

                    Question for the room For the Qwen2.5-7B strategy on d+2, the gross mean is 5.64 bp a day and about 95% of the days are active. With a one-way cost c = 5 bp per trade, what is the net mean?
                    Answer

                    About −13.36 bp a day. An active day pays 4c: entry and exit for the stock position and for the SPY hedge. The net mean is 5.64 − 0.95 × 4 × 5 ≈ −13.36 bp. +0.89 bp counts a single direction; −3.86 bp counts a single round trip. The Sharpe ratio falls from +0.63 to −1.48, and the break-even cost is 1.5 bp per trade.

                    Before costs, FinBERT earns +3.27 bp per day (\(t\) = +1.27), with 95% CI [−1.51; +8.93], not different from zero; LM earns −0.21 bp (\(t\) = −0.07). Qwen2.5-7B earns +5.64 bp per day (\(t\) = +2.26, \(p\) = 0.024), with CI [+0.95; +10.61]. As the best of three scores, its Holm p-value is 0.071, not significant.

                    Costs remove the gain. An active day pays \(4c\), and with active days on about 95% of days the net mean is \(5.64 - 0.95 \times 4 \times 5 \approx -13.36\) bp a day. The break-even cost of Qwen2.5-7B is 1.5 bp per trade, well below 5 bp, and its Sharpe ratio falls from +0.63 before costs to −1.48 after costs. The result supports association only: the large gains are concentrated in 2020–2021 and are not significant at 5% (Further Topics).

                    In brief

                    • Sum of daily excess returns (%), before costs and with 5 bp per trade
                    • FinBERT: +3.27 bp per day (\(t\) = +1.27), not different from zero
                      • CI 95% [\(-\)1.51; +8.93]; LM: \(-\)0.21 bp (\(t\) = \(-\)0.07)
                    • Qwen2.5-7B: +5.64 bp (\(t\) = +2.26, \(p\) = 0.024)
                      • CI 95% [+0.95; +10.61]; but it is the best of the three scores tested: Holm \(p\) across the three = 0.071, not significant
                      • Break-even cost +1.5 bp per trade, well below 5 bp: Sharpe +0.63 before, \(-\)1.48 after costs
                    • Type of evidence: association
                      • Only 2020–2021 has large gains, not significant at 5% (Further Topics)
                    MFM_ch15_news_signal

                    The Strategy: the Non-Tradable Day and Memorisation

                    Slide 45 in the PDF

                    The strategy on the non-tradable day \(d+1\) checks the timing. FinBERT earns −0.22 bp (\(t\) = −0.10) and Qwen +4.06 bp (\(t\) = +1.36). There is no significant gain even on that day: almost all of the reaction happens on day \(d\), which the date-only timestamps make impossible to trade.

                    Part of the Qwen edge may be memorised. The Qwen2.5 weights were released on 19 September 2024, and every headline in the sample, from 2010 to 2023, predates them. A headline that names an episode can trigger recall of what followed, unlike the Apple test of the previous section, which gave only the date. The memory test bounds what the model knows about dates; it does not bound what it knows about named events.

                    In brief

                    • Non-tradable day \(d+1\): FinBERT \(-\)0.22 bp (\(t\) = \(-\)0.10), Qwen +4.06 bp (\(t\) = +1.36)
                      • No significant gain even there: almost all of the reaction happens on day \(d\)
                    • Part of the Qwen edge may be memorised: the Qwen2.5 weights were released on 19 September 2024, after the 2010–2023 headlines
                      • A headline that names the episode can trigger recall, unlike the Apple test, which gave only the date
                    MFM_ch15_news_signal

                    Section summary

                    Headline scores explain the same-day move: +39.2 bp per standard deviation for FinBERT and +41.4 bp for Qwen, robust to two-way clustering, the wild bootstrap and Holm. The relation reflects in large part headlines that describe price moves. At the first safe horizon, \(d+2\), no slope survives the corrections, and the daily long-short strategy brings no net profit: the best score, Qwen2.5-7B, breaks even at 1.5 bp per trade, below the assumed cost of 5 bp.

                    Measurement Error in Sentiment Scores

                    A sentiment score is the output of a model, not the true tone of the text. When it enters a regression as a regressor, or when its mean is reported as a population quantity, the measurement error of the score passes into the estimates. This section derives the attenuation of a slope on a generated regressor, measures it for the chapter's three scores, and presents prediction-powered inference, which builds valid intervals from model scores and a small human-labelled sample.

                    Sentiment as a Generated Regressor: Classical Error

                    Slide 46 in the PDF

                    The question of the section is what happens to the slope and its interval when the regressor is a model output rather than the true tone. Let the true model be \(r = a + \beta s^* + \varepsilon\), where \(s^*\) is the true tone of the text. We observe only a model output \(\hat s = s^* + u\), where \(u\) is the measurement error.

                    Under classical error, \(u\) is independent of \(s^*\) and of \(\varepsilon\). The OLS slope of \(r\) on \(\hat s\) then converges to

                    \[\operatorname{plim} \hat b = \beta\lambda, \qquad \lambda = \frac{\operatorname{Var}(s^*)}{\operatorname{Var}(s^*) + \operatorname{Var}(u)} < 1,\]
                    (15.18)

                    where \(\lambda\) is the reliability of the score, the share of its variance that is signal. The slope is attenuated towards zero by the factor \(\lambda\).

                    Two consequences follow. The test of \(\beta = 0\) remains valid, because \(\beta\lambda = 0\) exactly when \(\beta = 0\). The magnitude, however, is understated: a slope estimated on a noisy score underestimates the effect of the true tone, and two scores with different reliabilities give different slopes even if they measure the same thing.

                    In brief

                    • The question of this section: what happens to the slope and its interval
                      • When the regressor is a model output, not the true tone?
                    • Model: \(r = a + \beta s^* + \varepsilon\) with true tone \(s^*\)
                      • We observe a model output \(\hat s = s^* + u\)
                      • Classical error (\(u\) independent of \(s^*\) and \(\varepsilon\)): \(\operatorname{plim} \hat b = \beta\lambda\), \(\lambda = \text{Var}(s^*)/[\text{Var}(s^*) + \text{Var}(u)] < 1\)
                      • \(\lambda\): the reliability of the score, the share of its variance that is signal
                      • The test of \(\beta = 0\) stays valid; magnitudes are attenuated towards 0

                    Standardised Score: Correcting by the Correlation with the True Tone

                    Slide 47 in the PDF

                    The regressions of the previous section use the standardised score \(z\), with \(\operatorname{Var}(z) = 1\). Assume the error is non-differential: \(E[r \mid s^*, \hat s] = E[r \mid s^*]\), so that the score carries no information about returns beyond the true tone. Then \(\operatorname{Cov}(r, z) = \beta\,\operatorname{Cov}(s^*, z)\), and since \(\operatorname{Var}(z) = 1\),

                    \[b_z = \operatorname{Cov}(r, z) = \beta\,\sigma_{s^*}\,\rho(s^*, \hat s).\]
                    (15.19)

                    The slope on \(z\) is the effect of one standard deviation of the true tone, \(\beta\sigma_{s^*}\), multiplied by the correlation between truth and score. Dividing the estimated slope by \(\rho(s^*, \hat s)\) recovers the effect of one standard deviation of the true tone.

                    A binary label is not classical error. If the regressor is a dummy obtained from a classifier, misclassification shrinks the slope by a different factor (Aigner, 1973); the formula is in the Appendix. The econometrics of AI-generated variables is an active field: Battaglia et al. (2024) propose bias correction and joint estimation, and Ludwig, Mullainathan & Rambachan (2024) distinguish the prediction and estimation uses of LLMs.

                    In brief

                    • Standardised score \(z\), non-differential error: \(\text{Cov}(r, z) = \beta\,\text{Cov}(s^*, z)\)
                      • Non-differential error: \(E[r \mid s^*, \hat s] = E[r \mid s^*]\)
                      • Since \(\text{Var}(z) = 1\), the slope is \(b_z = \text{Cov}(r, z) = \beta\,\sigma_{s^*}\,\rho(s^*, \hat s)\)
                      • Slope on \(z\) = (effect of one standard deviation of \(s^*\)) \(\times\ \rho(s^*, \hat s)\): divide by the correlation of truth and score
                      • A binary label is not classical error: misclassification shrinks a dummy slope by a different factor (Aigner (1973); Appendix and Seminar 15, A1)
                    • Econometrics of AI-generated variables

                    How Much Attenuation in Our Slopes?

                    Slide 48 in the PDF

                    The correction (15.19) needs \(\rho(s^*, \hat s)\), the correlation between the score and the true tone. It is measured on the Twitter validation set, as the correlation of the net score with the human label coded \(-1, 0, 1\), one headline at a time: 0.29 for LM, 0.58 for FinBERT and 0.71 for Qwen2.5-7B.

                    Dividing the same-day slopes by these correlations gives the effect of one standard deviation of the true tone: 15.5/0.29 ≈ 54 bp for LM, 39.2/0.58 ≈ 67 bp for FinBERT and 41.4/0.71 ≈ 58 bp for Qwen. After the correction the three methods agree much more than before. The small LM slope is then mostly measurement error, provided the assumptions discussed in Further Topics hold.

                    A second approach uses only the FNSPID scores. Treating the three daily scores as noisy measures of a common factor \(f\), the loading \(\ell_j = \operatorname{corr}(z_j, f)\) of score \(j\) is 0.53 for LM, 0.85 for FinBERT and 0.78 for Qwen; the factor model and the instrumental-variables check are in the Appendix. The effect of one standard deviation of \(f\), \(b/\ell_j\), is then 29 bp (LM), 46 bp (FinBERT) and 53 bp (Qwen). The two approaches agree that the corrected effects are closer to each other than the raw slopes, though not on the exact values.

                    In brief

                    Score\(\rho(s^*, \hat s)\), Twitter\(b\) on day \(d\) (bp)\(b/\rho\) (bp)\(\ell_j\), one-factor loading
                    LM0.2915.5540.53
                    FinBERT0.5839.2670.85
                    Qwen2.5-7B0.7141.4580.78
                    • \(\rho\): correlation of the net score with the human label (\(-1, 0, 1\)), one headline at a time
                      • \(b/\rho\): same-day effect of one standard deviation of true tone
                      • After correction the three methods agree much more: the small LM slope is mostly measurement error, if the assumptions in Further Topics hold
                    • \(\ell_j\): loading of score \(j\) on the common factor \(f\) of the three daily scores
                      • \(\ell_j = \text{corr}(z_j, f)\) (factor model and instruments: Appendix)
                      • Effect of one standard deviation of \(f\), \(b/\ell_j\): 29 bp (LM), 46 bp (FinBERT), 53 bp (Qwen)

                    Valid Inference with Model Labels: Prediction-Powered Inference

                    Slide 49 in the PDF

                    Suppose we want the population mean of the human label, \(\theta = E[Y]\), for example the mean tone of a news corpus. We have \(n\) texts labelled by humans, \((X_j, Y_j)\), and \(N\) unlabelled texts \(\tilde X_i\), independent random samples of the same population, and the score \(f(\cdot)\) of a fixed model trained on other data. The mean of \(f\) over the unlabelled texts is precise but biased if the model is biased; the mean of the human labels is unbiased but imprecise if \(n\) is small.

                    Prediction-powered inference (PPI) (Angelopoulos et al., 2023) combines the two:

                    \[\hat\theta_{\text{PPI}} = \frac{1}{N}\sum_{i=1}^{N} f(\tilde X_i) - \frac{1}{n}\sum_{j=1}^{n}\big(f(X_j) - Y_j\big).\]
                    (15.20)

                    The first term is the mean prediction over the \(N\) unlabelled texts. The second term is the mean bias \(f - Y\), estimated on the labelled texts and subtracted. Its variance is estimated by

                    \[\widehat{\operatorname{Var}}(\hat\theta_{\text{PPI}}) = \frac{\widehat{\operatorname{Var}}(f)}{N} + \frac{\widehat{\operatorname{Var}}(f - Y)}{n},\]
                    (15.21)

                    which gives an asymptotically valid confidence interval under this sampling design, whatever the quality of \(f\). A good model makes \(\operatorname{Var}(f - Y)\) small and the interval narrow; a poor model makes the interval wide but does not invalidate it. Extensions to regressions are in Further Topics.

                    In brief

                    • Data: \(n\) texts labelled by humans (\(X_j\), \(Y_j\)) and \(N\) unlabelled texts \(\tilde X_i\)
                      • Independent random samples of one population
                    • Target: \(\theta = E[Y]\), the population mean of the human label
                      • \(f(\cdot)\) is the score of a fixed model, trained on other data
                    • PPI (prediction-powered inference, Angelopoulos et al. (2023)): \(\hat\theta_{\text{PPI}} = \frac{1}{N}\sum_{i=1}^{N} f(\tilde X_i) - \frac{1}{n}\sum_{j=1}^{n} \bigl(f(X_j) - Y_j\bigr)\)
                      • First term: the mean prediction over the \(N\) unlabelled texts
                      • Second term: the mean bias \(f - Y\), estimated on the labelled texts and subtracted
                    • \(\widehat{\text{Var}}(\hat\theta_{\text{PPI}}) = \widehat{\text{Var}}(f)/N + \widehat{\text{Var}}(f - Y)/n\)
                      • An asymptotically valid CI under this sampling design, whatever the quality of \(f\)

                    PPI on Twitter Headlines: the Bias Correction Is the Main Gain (1/2)

                    Slide 50 in the PDF

                    The simulation treats the 2,388 Twitter validation headlines as the population and estimates their mean human tone. The left panel shows the mean width of the 95% interval and the right panel its empirical coverage, both against the number \(n\) of human-labelled headlines (100, 200, 400 and 800). The three lines are human labels only, PPI with FinBERT scores and PPI with Qwen2.5-7B scores; the dashed line in the right panel marks 95%.

                    In the interactive version, the slider sets \(n\), and a third panel shows the distribution of the estimates across replications. The ▶ button raises \(n\) step by step.

                      Question for the room Keep FinBERT fixed and add unlabelled headlines from the same population. Does this remove the difference between the mean FinBERT score (−0.014) and the mean human tone (0.054)?
                      Answer

                      No: the uncertainty of the mean score falls, but the bias remains. More scores reduce sampling error; they do not correct the bias of the model. PPI estimates the mean difference f − Y on a labelled sample and subtracts it from the mean score. At n = 100, intervals with nominal level 95% have empirical coverage of 94.3% in the lecture and 94.4% in the browser; the mean width is 0.204 in the lecture and 0.203 in the browser, against 0.229 with human labels only.

                      Adding unlabelled headlines while keeping FinBERT fixed does not remove the difference between the mean FinBERT score, −0.014, and the mean human tone, 0.054: more scores reduce sampling error, not the bias of the model. PPI estimates the mean difference \(f - Y\) on the labelled sample and subtracts it, as in (15.20). At \(n\) = 100, the intervals with nominal level 95% have empirical coverage of 94.3%, and their mean width is 0.204, against 0.229 with human labels only.

                      PPI on Twitter Headlines: the Bias Correction Is the Main Gain (2/2)

                      Slide 51 in the PDF

                      The simulation draws, in each of 2,000 replications, \(n\) labelled and \(2{,}388 - n\) unlabelled headlines independently and with replacement from the 2,388 validation headlines. The target is the mean human net tone, with labels coded \(-1, 0, 1\); its true value in this population is \(\theta = 0.054\).

                      The plain means of the model scores are biased: −0.014 for FinBERT, which has the wrong sign, and +0.101 for Qwen2.5-7B. No sample size fixes this, because the bias is a property of the model. PPI keeps nominal coverage, 94.3% at \(n\) = 100. Its interval is narrower than the one from labels only, 0.204 against 0.229 for FinBERT at \(n\) = 100, but the gain is modest: the variance reduction \(1 - \operatorname{Var}(f - Y)/\operatorname{Var}(Y)\) is 24% for FinBERT and 27% for Qwen, because the models are noisy.

                      At \(n\) = 800 the PPI interval is wider than the one from labels only, 0.087 against 0.081. With only \(2{,}388 - 800\) unlabelled texts, the term \(\operatorname{Var}(f)/N\) in (15.21) is no longer negligible. For the return regressions, the same labelled sample allows both OLS moments to be rectified (Egami et al., 2023). The result supports a comparison of procedures: PPI, human labels only and the plain mean of the scores.

                      In brief

                      • Target: mean human net tone (\(-1, 0, 1\))
                        • The 2 388 validation headlines act as the population
                        • 2,000 replications of \(n\) labelled and 2 388 \(- n\) unlabelled headlines drawn independently, with replacement
                      • True mean tone \(\theta\) = 0.054
                        • Plain model means are biased: FinBERT \(-\)0.014 (wrong sign), Qwen2.5-7B +0.101
                        • No sample size fixes this
                      • PPI keeps nominal coverage (94.3% at \(n\) = 100)
                        • Mean width at \(n\) = 100: 0.229 with labels only, 0.204 with PPI (FinBERT)
                        • Variance reduction \(1 - \text{Var}(f - Y)/\text{Var}(Y)\): 24% (FinBERT), 27% (Qwen)
                        • Modest because the models are noisy
                      • At \(n\) = 800 PPI is wider (0.087 vs 0.081)
                        • With only 2 388 \(-\) 800 unlabelled texts, \(\text{Var}(f)/N\) is no longer negligible
                        • For the return regressions, the same labelled sample allows rectifying both OLS moments (Egami et al. (2023))
                      • Type of evidence: comparison of procedures (PPI, human labels only, the mean of the scores)
                      MFM_ch15_valid_inference

                      Section summary

                      A model score is a measurement with error. Under classical error the slope on the score is attenuated by the reliability (15.18), and for a standardised score by the correlation with the true tone (15.19); corrected in this way, the same-day effects of the three scores become much closer. PPI (15.20) uses a small human-labelled sample to remove the bias of the mean score and gives valid intervals whatever the quality of the model; the main gain is the bias correction, not a narrower interval.

                      Case Studies: ChatGPT and the Colour of Finance Words

                      Two recent papers in the Journal of Financial Economics address the chapter's question with larger samples and more precise data. Lopez-Lira & Tang (2026) score headlines with GPT-4 after its training cutoff and with exact publication times. Garcia, Hu & Rohrer (2023) learn word lists from price reactions instead of choosing them by judgement. Both are replicated in part on the chapter's FNSPID data.

                      Research Question: Does an LLM Read News as the Market Does?

                      Slide 52 in the PDF

                      Lopez-Lira & Tang (2026) ask whether a general-purpose LLM, with no explicit financial training, can tell whether a headline is good or bad for the stock, and whether its answer predicts returns. They separate two quantities. The initial reaction is the price move right after the news; it is not tradable, and it measures whether the LLM understands the news. The drift is the return after the reaction, in the direction of the news; it is tradable, and it measures the underreaction of the market.

                      The design is the same zero-shot scoring of headlines as in the chapter's FNSPID test, with two differences: the news is published after the model's training cutoff, and its publication times are exact. Both remove problems identified in the previous sections, memorisation and date-only timestamps.

                      The paper matters because it turns the LLM into a measuring instrument for how fast prices absorb news. The authors' model also predicts that only LLMs above a quality threshold predict returns and that wide adoption of such models should erode the predictability. The method of the study is detailed in Further Topics.

                      In brief

                      • Lopez-Lira & Tang (2026) (Journal of Financial Economics) ask:
                        • Can a general-purpose LLM, with no explicit financial training, tell whether a headline is good or bad for the stock?
                      • Does its answer predict returns?
                      • Initial reaction: the price move right after the news
                        • Not tradable: it measures whether the LLM understands the news
                      • Drift: the return after the reaction, in the direction of the news
                        • Tradable: it measures underreaction of the market
                      • Versus our FNSPID test: the same zero-shot scoring of headlines
                        • But on news after the model's training cutoff and with exact publication times
                      • Why it matters: the LLM becomes a measuring instrument for how fast prices absorb news
                        • Their model: only LLMs above a quality threshold predict returns, and wide adoption should erode the predictability

                      GPT-4 Gets the Reaction Right and Earns the Drift

                      Slide 53 in the PDF

                      The table reports, for four of the models, the hit rate of the long-short portfolio on overnight news, the share of days on which the portfolio return is positive, for the initial reaction and for the drift, together with the mean drift return per day and its annualised Sharpe ratio \(SR\). GPT-4 gets the direction of the reaction right on 93% of days, like GPT-3.5 (Generative Pre-trained Transformer 3), but it earns the largest drift: 0.34% per day, with \(SR\) 2.97, against 0.29% and 1.66 for GPT-3.5. FinBERT reaches a 90% hit rate on the reaction, but its drift is −0.07% per day, with \(SR\) −0.33. GPT-1 (Generative Pre-trained Transformer 1) is near chance on the reaction, at 0.53.

                      The legs of the GPT-4 portfolio are asymmetric. The short leg earns 0.26% per day (\(SR\) 2.01) and the long leg 0.08% (\(SR\) 0.78): the market underreacts mainly to bad news. In the regression on overnight news, the coefficient of the GPT-4 score is \(\hat\gamma = 1.326\) (\(t\) = 26.52) for the reaction and 0.161 (\(t\) = 6.59) for the drift. Adding the news vendor's own sentiment score leaves GPT-4 at 0.158 (\(t\) = 5.27) and gives the vendor 0.010 (\(t\) = 0.15). The interaction with small stocks is 0.404 (\(t\) = 4.55): the drift is concentrated where arbitrage is costly.

                      The result supports association. It is also not stable over time: the GPT-4 Sharpe ratio falls from 6.54 in the fourth quarter of 2021 to 1.22 in January–May 2024, as shown in Further Topics.

                      In brief

                      ModelHit rate, reactionHit rate, driftDrift, % per day\(SR\)
                      GPT-40.930.580.342.97
                      GPT-3.50.930.560.291.66
                      FinBERT0.900.48\(-0.07\)\(-0.33\)
                      GPT-10.530.48\(-0.07\)\(-0.64\)
                      • Overnight news, long-short portfolios: GPT-4 gets the direction of the reaction right on 93% of days
                        • Like GPT-3.5 (version 3.5 of GPT), but it earns the largest drift
                        • Short leg 0.26% per day (\(SR\) 2.01), long leg 0.08% (\(SR\) 0.78): the market underreacts mainly to bad news
                      • Regression of returns on the GPT-4 score, overnight news: \(\hat\gamma = 1.326\) (\(t = 26.52\)) for the reaction, \(0.161\) (\(t = 6.59\)) for the drift
                        • Adding the vendor's sentiment score leaves GPT-4 at 0.158 (\(t = 5.27\)) and the vendor at 0.010 (\(t = 0.15\))
                        • Small stocks: interaction coefficient 0.404 (\(t = 4.55\)): the drift is concentrated where arbitrage is costly
                      • Type of evidence: association
                        • The GPT-4 Sharpe ratio falls from 6.54 (2021 Q4) to 1.22 (January–May 2024): Further Topics

                      Which Models Read the News Right, and Which Earn the Drift (1/2)

                      Slide 54 in the PDF

                      The chart reports the results of Lopez-Lira & Tang (2026) for twelve language models scored with the same design: long-short portfolios on overnight news from October 2021 to May 2024. In the left panel, each model has two horizontal bars, the hit rate of the initial reaction and the hit rate of the drift, in percent, with a dashed line at 50%. The right panel shows the annualised Sharpe ratio of the drift, green when positive and red when negative. The models include BART (Bidirectional and Auto-Regressive Transformer), an encoder–decoder language model, DistilBart, its compressed (distilled) version, and the Llama2 open-weight LLMs released by Meta.

                      In the interactive version, the slider sets a threshold on the hit rate of the initial reaction, from 50% to 95%, and highlights the models above it. The ▶ button raises the threshold step by step.

                        Question for the room FinBERT has a 90% hit rate for the initial reaction and a drift Sharpe ratio of −0.33. Which distinction explains these results?
                        Answer

                        Identifying the initial reaction does not guarantee profitable subsequent returns. The initial reaction and the drift concern different windows: information already reflected in the price can identify the reaction without offering a later gain. GPT-4 and GPT-3.5 have the highest drift Sharpe ratios, 2.97 and 1.66, but they are not the only positive ones: DistilBart has 1.26, BART-Large 1.05. The models with hit rates of 53–66% have negative Sharpe ratios.

                        Identifying the initial reaction does not guarantee profitable later returns. FinBERT has a hit rate of 90% for the reaction and a drift Sharpe ratio of −0.33: information already reflected in the price can identify the reaction without offering a later gain. GPT-4 and GPT-3.5 have the highest drift Sharpe ratios, 2.97 and 1.66, but they are not the only positive ones: DistilBart has 1.26 and BART-Large 1.05. The models with reaction hit rates of 53–66% have negative Sharpe ratios.

                        In brief

                        BART: Bidirectional and Auto-Regressive Transformer, an encoder–decoder language model; DistilBart: a compressed (distilled) BART; Llama2: open-weight LLMs released by Meta MFM_ch15_llt_case

                        Which Models Read the News Right, and Which Earn the Drift (2/2)

                        Slide 55 in the PDF

                        The twelve models fall into groups. GPT-1, GPT-2 (Generative Pre-trained Transformer 2), BERT and Llama2-7b get the reaction right on only 53–66% of days, and their drift Sharpe ratios are negative, from −0.64 to −0.11. Seven models reach 86–93% on the reaction, but only GPT-4 and GPT-3.5 combine it with a large drift Sharpe ratio. This pattern is consistent with the threshold in the authors' model: a model must understand the news well before its signal earns the drift.

                        FinBERT gets the reaction right but has a negative drift. It gives a direction for only 25% of overnight headlines, and reading what is already in the opening price does not earn the drift.

                        The legs need to be read alongside the Sharpe ratios. The average number of stocks held in the short leg is 2 for BERT-Large, whose Sharpe ratio is 1.05, 6 for GPT-3.5 and 20 for GPT-4. A Sharpe ratio built on a handful of short positions is fragile.

                        In brief

                        • Long-short portfolios on overnight news, twelve models, same design
                        • Threshold: GPT-1 (Generative Pre-trained Transformer 1), GPT-2 (the first GPT versions), BERT and Llama2-7b get the reaction right on only 53–66% of days
                          • They have negative drift Sharpe ratios (\(-0.64\) to \(-0.11\))
                          • Seven models reach 86–93%, but only GPT-4 (2.97) and GPT-3.5 (1.66) pair it with a large drift Sharpe ratio; for every other model it is at most 1.26
                        • FinBERT gets the reaction right but has a negative drift
                          • It gives a direction for only 25% of overnight headlines
                          • Reading what is already in the opening price does not earn the drift
                        • Read the legs: stocks held on average in the short leg
                          • 2 stocks for BERT-Large (Sharpe 1.05), 6 for GPT-3.5, 20 for GPT-4
                          • A Sharpe ratio from a handful of shorts is fragile
                        MFM_ch15_llt_case

                        Replication in This Course, Limitations and an Open Extension

                        Slide 56 in the PDF

                        The Qwen2.5-7B strategy of this chapter on the 16 FNSPID stocks earns +5.64 bp per day before costs, against 34 bp for GPT-4 in the paper. Part C3 of Seminar 15 replicates the paper's design on the course data: the published prompt, temperature 0, scores in \(\{-1, 0, 1\}\), the rule of at least two firms per leg and the same regression, on FNSPID headlines and the adjusted daily closes of the course data. The data impose three limits:

                        1. date-only timestamps, so the first safe return is \(d+2\) rather than the next open;
                        2. 16 large stocks;
                        3. headlines that predate the model.

                        The authors state their own limits. At 20 bp per round trip the strategy loses money, so it suits low-cost traders such as market makers; the chapter's 5 bp per one-way trade corresponds to 10 bp per round trip. The prompts and embedding models were not optimised, so their comparisons are lower bounds.

                        An open extension follows from the paper's model. On headlines published after the release of an open-weight model, does the drift survive, and does its Sharpe ratio keep falling as the model of the authors predicts?

                        In brief

                        • The Qwen2.5-7B strategy of this chapter on 16 FNSPID stocks: +5.64 bp per day
                          • Against 34 bp for GPT-4
                          • Replication (Seminar 15, C3): the published prompt, temperature 0, scores \(\{-1, 0, 1\}\), the two-firm rule (a leg only with at least two firms) and the same regression, on FNSPID headlines and the adjusted daily closes of the course data
                          • Our data limits: date-only timestamps (the first safe return is \(d+2\), not the next open), 16 large stocks, headlines that predate the model
                        • Limits stated by the authors: at 20 bp per round trip the strategy loses money
                          • So it suits low-cost traders such as market makers
                          • Our 5 bp per one-way trade is 10 bp per round trip
                          • Prompts and embedding models were not optimised, so the comparisons are lower bounds
                        • Open extension, on headlines published after the release of an open-weight model
                          • does the drift survive?
                          • does its Sharpe ratio keep falling as the authors' model predicts?

                        A Second Case Study: The Colour of Finance Words

                        Slide 57 in the PDF

                        Garcia, Hu & Rohrer (2023) ask whether a word list chosen by price reactions can compete with lists chosen by researchers, such as Loughran–McDonald. Their corpus is the transcripts of 85,530 earnings calls, the calls in which firms present quarterly results, of 3,229 firms from 13 October 2005 to 7 October 2020. The label of each text is the stock's return minus the market return from the close of day \(t-1\) to the close of day \(t+2\).

                        In the paper's main result, lists learned on the calls before 2016 are evaluated on 2016–2020, together with the paper's control variables. The adjusted \(R^2\) is 2.1% with the LM words that are not in the learned (ML, machine learning) lists, 4.6% with the ML words that are not in LM, and 1.7% with the controls only. The lists learned on the full sample share only 18 positive and 30 negative words with LM.

                        The approach keeps the transparency of a word list while letting the market, rather than the researcher's judgement, decide the colour of each word. It is therefore the simple benchmark against which FinBERT and LLMs should be compared on the same texts.

                        In brief

                        • Garcia, Hu & Rohrer (2023) (Journal of Financial Economics), the question:
                          • Can a word list chosen by price reactions compete with lists chosen by researchers, such as Loughran–McDonald (LM)?
                        • The paper's corpus: transcripts of 85 530 earnings calls (the calls in which firms present quarterly results), 3 229 firms, 13 Oct 2005 – 7 Oct 2020
                        • The label of each text: the stock's return minus the market return, from the close of day \(t-1\) to the close of day \(t+2\)
                        • The paper's main result: lists learned on the calls before 2016, evaluated on 2016–2020, with the paper's control variables
                          • Adjusted \(R^2\): 2.1% with the LM words not in the ML (Machine Learning) lists, 4.6% with the ML words not in LM
                          • With the controls only: 1.7%
                          • The lists learned on the paper's full sample share only 18 positive and 30 negative words with LM
                        • Why it matters: the list stays transparent, but the colour of each word comes from the market, not from the researcher's judgement
                          • It is the simple benchmark against which FinBERT and LLMs should be compared on the same texts

                        Word Lists Learned from Returns, on the FNSPID Headlines

                        Slide 58 in the PDF

                        The chart shows the course replication on the FNSPID headlines. The lists are built on 2009–2019, and \(R^2\) and slopes are estimated on 2020–2023. The left panel reports the \(R^2\), in percent, of the 4-day excess return on each set of words and on the two language-model scores: LM lists in orange, lists learned from returns in turquoise, language models in blue. The right panel shows the excess return per one standard deviation of the word share, in percent, for the positive and negative lists, with 95% intervals based on Driscoll–Kraay standard errors.

                        In the interactive version, the slider sets the cut-off on \(|D^+|\), the net share of subsamples in which a word has a positive or negative loading, from 50% to 95%; the ▶ button raises the cut-off step by step.

                          Question for the room At |D+| ≥ 80%, the ML words that are not in LM give R² = 0.77% on the 2020–2023 texts, against 0.20% for all LM words. What does this comparison support?
                          Answer

                          A stronger association on the 2020–2023 texts, without establishing a tradable forecast. The lists are learned on 2009–2019, but the slopes and R² are estimated on 2020–2023, and the 4-day return window may include the move reported by the headline: many ML words ("jumped", "slumped", "gainers") describe the price move itself. The R² difference is 0.58 percentage points, 95% CI [0.14; 1.02] (bootstrap over blocks of 10 days); it is not a strategy return. FinBERT (1.49%) and Qwen2.5-7B (1.73%) explain more than any list.

                          At the cut-off of 80%, the ML words that are not in LM give \(R^2\) = 0.77% on the 2020–2023 texts, against 0.20% for all LM words. This is a stronger association on these texts, not a tradable forecast: the slopes and \(R^2\) are estimated on the same 2020–2023 texts, and the 4-day return window may include the move the headline reports. Many ML words, such as "jumped", "slumped" or "gainers", describe the price move itself. FinBERT (1.49%) and Qwen2.5-7B (1.73%) explain more than any list.

                          In brief

                          ML (machine learning): the lists learned from returns; the lists are built on 2009–2019, \(R^2\) and slopes are estimated on 2020–2023 MFM_ch15_colour_words

                          Method: Robust MNIR and What the Course Replication Changes

                          Slide 59 in the PDF

                          The method is multinomial inverse regression (MNIR) (Taddy, 2013), described in the Appendix. It models the count of word \(j\) in text \(i\) as Poisson with mean

                          \[m_i\, e^{\alpha_j + \varphi_j v_i},\]
                          (15.22)

                          where \(m_i\) is the number of words of the text and \(v_i\) the standardised return. A positive \(\varphi_j\) means that the word is more frequent after high returns.

                          The robust step repeats MNIR on 500 subsamples of 5,000 texts each, drawn without replacement. For each word, \(D^+_j\) is the share of subsamples with \(\varphi_j > 0\) minus the share with \(\varphi_j < 0\). The positive list contains the words with \(D^+_j \ge 80\%\) and the negative list those with \(D^+_j \le -80\%\). The score of a text is the number of list words divided by the number of words, scaled to unit standard deviation.

                          The course replication changes the unit and the method. A text is the set of all FNSPID headlines of one stock on one day, 18,282 stock-days. Training uses October 2009 to December 2019, without the texts whose return window reaches into 2020 (10,286 texts), and testing uses 2020–2023 (7,980 texts). The vocabulary of 15,376 words and the winsorization cut-offs come from the training texts only. The sign of \(\varphi_j\) is obtained by a score-test approximation of the selection by the Akaike information criterion (AIC), \(z^2 > 2\), instead of the gamma-lasso penalty. The paper's control variables are not used, and the names of the 16 firms are removed instead of applying the proper-noun filter.

                          In brief

                          • MNIR (multinomial inverse regression, Taddy (2013), Appendix): the count of word \(j\) in text \(i\) is Poisson, with mean \(m_i e^{\alpha_j+\varphi_j v_i}\)
                            • \(m_i\): the number of words of the text; \(v_i\): the standardised return
                          • The robust step: MNIR on 500 subsamples of 5 000 texts each, drawn without replacement
                            • \(D^+_j\): the share of subsamples with \(\varphi_j > 0\) minus the share with \(\varphi_j < 0\)
                            • Positive list: \(D^+_j \ge 80\%\); negative list: \(D^+_j \le -80\%\)
                          • The score of a text: the number of list words divided by the number of words, scaled to unit standard deviation
                          • The course replication, on the chapter's data:
                            • The text: all FNSPID headlines of one stock on one day, 18 282 stock-days
                            • Training: Oct 2009 – Dec 2019, without the texts whose return window reaches into 2020 (10 286 texts); test: 2020–2023 (7 980 texts)
                            • Vocabulary (15 376 words) and winsorization cut-offs: from the training texts only
                            • The sign of \(\varphi_j\): a score-test approximation of the AIC selection (Akaike information criterion, \(z^2 > 2\)), instead of the gamma-lasso penalty
                            • No control variables of the paper; the names of the 16 firms are removed instead of the proper-noun filter

                          The Result: Words Chosen by the Market Explain More than LM

                          Slide 60 in the PDF

                          On the 2020–2023 test texts, with slopes estimated on the same period, the \(R^2\) is 0.20% with all LM words and 0.77% with the 94 ML words that are not in LM. The difference is +0.58 percentage points, with a 95% CI of [0.14; 1.02], from a bootstrap over blocks of 10 consecutive days with 1,999 draws and the learned lists held fixed. For cut-offs from 50% to 95%, the \(R^2\) of the ML lists lies between 0.44% and 1.02%.

                          The slopes are reported per one standard deviation of the word share, with \(t\) statistics from Driscoll–Kraay standard errors, which are HAC (heteroskedasticity and autocorrelation consistent) over days. The positive ML list has a slope of +0.31% (\(t\) = 4.45) and the negative ML list −0.24% (\(t\) = −3.98). The positive LM words are not significantly different from zero, +0.04% (\(t\) = 0.81). The negative LM words have −0.19% (\(t\) = −3.29), and remain significant next to the ML lists, at −0.17% (\(t\) = −2.90). FinBERT (1.49%) and Qwen2.5-7B (1.73%) explain more than any list; Qwen exceeds the ML lists by +0.96 percentage points, with 95% CI [0.20; 1.73].

                          The result supports a comparison of procedures on an association within the same 4-day window, not a forecast. Many ML words describe the price move itself ("jumped", "slumped", "gainers", "losers"), so the headline may follow the return. Only 1 positive and 5 negative words are shared with LM: "positive" on one side, and "claims", "downgrades", "weak", "worse" and "worst" on the other. Qwen may already know what followed the 2020–2023 news (Look-Ahead Bias).

                          In brief

                          • \(R^2\) on the 2020–2023 test texts, slopes estimated on the same period: 0.20% with all LM words, 0.77% with the 94 ML words that are not in LM
                            • The difference: +0.58 percentage points, 95% CI [0.14; 1.02] (bootstrap over blocks of 10 consecutive days, 1 999 draws, learned lists held fixed)
                            • For cut-offs on \(D^+_j\) from 50% to 95% (instead of 80%), the \(R^2\) of the ML lists is between 0.44% and 1.02%
                          • One standard deviation more in the share of ML words (\(t\) with Driscoll–Kraay standard errors, HAC over days)
                            • Positive: +0.31% (\(t = 4.45\)); negative: \(-\)0.24% (\(t = -3.98\))
                          • The coefficient of the positive LM words is not significantly different from zero: +0.04% (\(t = 0.81\))
                            • Negative LM words: \(-\)0.19% (\(t = -3.29\)), significant also next to the ML lists: \(-\)0.17% (\(t = -2.90\))
                          • FinBERT (1.49%) and Qwen2.5-7B (1.73%) explain more than any list
                            • Qwen against the ML lists: +0.96 percentage points, 95% CI [0.20; 1.73]
                          • Type of evidence: a comparison of procedures, on an association within the same 4-day window, not a forecast
                            • Many ML words describe the price move itself (“jumped”, “slumped”, “gainers”, “losers”): the headline may follow the return
                          • Limits: only 1 positive and 5 negative words shared with LM (positive; claims, downgrades, weak, worse, worst)

                          Section summary

                          GPT-4, scored on news published after its training cutoff, gets the initial reaction right on 93% of days and earns a drift with a Sharpe ratio of 2.97 over October 2021–May 2024, mostly from bad news and small stocks; models below a quality threshold earn nothing, and the Sharpe ratio falls over time. Word lists learned from returns explain more of the 4-day return of FNSPID headlines than LM, but less than FinBERT and Qwen2.5-7B, and much of what they capture is headlines that describe price moves.

                          AI for Scientific Discovery

                          LLMs can propose hypotheses, write code and run experiments. This section asks what inference remains valid when an assistant proposes the signal, the prompt, the threshold and the holding period, applies the answer to refinements of the chapter's strategy and tests the resulting family of strategies with a specification curve.

                          An Open Question: Can LLM-Generated Hypotheses Be Trusted?

                          Slide 61 in the PDF

                          Ludwig & Mullainathan (2024) treat hypothesis generation as a statistical procedure: a model trained on data reveals a pattern, humans name it, and fresh data test it. The procedure is valid because the test data are not used to generate the hypothesis.

                          An LLM breaks this separation in a specific way. Lopez-Lira, Tang & Zhu (2025) show that, before its knowledge cutoff, an LLM can recall outcomes. A hypothesis it proposes about that period is therefore not independent of the data that will test it.

                          The open question is what the family of tests is, and which inference stays valid, when an assistant proposes the signal, the prompt, the threshold and the holding period. Agents that run the experiments themselves (Lu et al., 2024) make the family both larger and less visible.

                          In brief

                          • Ludwig & Mullainathan (2024): hypothesis generation as a statistical procedure
                            • A model trained on data reveals a pattern, humans name it, fresh data test it
                          • Lopez-Lira, Tang & Zhu (2025): before its knowledge cutoff an LLM can recall outcomes
                            • So a hypothesis it proposes about that period is not independent of the data that will test it
                          • Open: when an assistant proposes the signal, the prompt, the threshold and the holding period
                            • what is the family of tests?
                            • which inference stays valid?
                          • Agents that run the experiments themselves (Lu et al. (2024)) make the family larger and less visible

                          The Family of Tests: FWER and FDR

                          Slide 62 in the PDF

                          Formally, the candidate strategies \(j = 1, \dots, M\) define the null hypotheses \(H_{0,j}: E[r^{(j)}_t] = 0\). Two error rates can be controlled. The FWER, controlled by Holm (15.14), is the probability of any false rejection. The false discovery rate (FDR) is the expected share of false rejections among all rejections (Benjamini & Hochberg, 1995).

                          The Benjamini–Hochberg (BH) procedure at level \(q\) rejects the hypotheses with \(p_{(1)}, \dots, p_{(k^*)}\), where

                          \[k^* = \max\Big\{k : p_{(k)} \le \frac{k\,q}{M}\Big\}.\]
                          (15.23)

                          The count \(M\) must include every variant the assistant proposed, including those discarded. If the hypotheses were built on the test outcomes, the p-values need untouched test data or selective (sequential) inference.

                          In brief

                          • Formally: \(H_{0,j}: E[r^{(j)}_t] = 0\) for candidate strategies \(j = 1, \dots, M\)
                            • Control the FWER (Holm) or the FDR (Benjamini & Hochberg (1995))
                            • FDR: expected share of false rejections among all rejections
                            • BH (the Benjamini–Hochberg procedure) at level \(q\): reject the hypotheses with \(p_{(1)}, \dots, p_{(k^*)}\), where \(k^* = \max\{k : p_{(k)} \le k q / M\}\)
                            • \(M\) counts every variant the assistant proposed, including those discarded; if the hypotheses were built on the test outcomes, the \(p\)-values need untouched test data or selective (sequential) inference

                          The Discovery Loop and What the Researcher Must Check

                          Slide 63 in the PDF

                          The research loop has six steps: literature, hypotheses, data and code, robustness checks fixed in advance, critique and report. An assistant such as Claude, ChatGPT, Gemini or GitHub Copilot can draft each step, but the researcher verifies each one; reusable prompts for the steps are in the Appendix. Four risks need a specific test.

                          Memorisation: the model may know what followed the headline. The test uses news published after the release of the weights, anonymised text as a diagnostic (Glasserman & Lin, 2023) and the memory test of this chapter. Forking paths: the prompt, threshold and horizon may be tuned on the test period. The test is a pre-registered grid with a stated error rate, Holm for the FWER or BH for the FDR under independence or positive dependence; the rule \(|t| > 3\) (Harvey, Liu & Zhu, 2016) is a heuristic, not an error-rate guarantee.

                          HARKing, hypothesising after the results are known, produces a plausible mechanism written after the result, with references that may not exist. The test is to state the mechanism and its falsifier before the test and to resolve every DOI (digital object identifier). Timing and agents that run code bring date-only timestamps and silent changes of data, windows or seeds. The tests are the first tradable return on \(d+2\), read-only data, logged tool calls and the diff of every script before a result is accepted.

                          In brief

                          • Literature, hypotheses, data and code, robustness fixed in advance, critique, report
                            • An assistant (Claude, ChatGPT, Gemini, GitHub Copilot) drafts each step
                            • The researcher verifies each one (reusable prompts: Appendix)
                          • Memorisation: the model knows what followed the headline
                            • Test: news after the release of the weights; anonymised text as a diagnostic (Glasserman & Lin (2023)); the memory test of this chapter
                          • Forking paths: prompt, threshold, horizon tuned on the test period
                            • Test: a pre-registered grid and a stated error rate: Holm (FWER) or BH (FDR, under independence or positive dependence); \(|t| > 3\) (Harvey, Liu & Zhu (2016)) is a heuristic, not an error-rate guarantee
                          • HARKing (hypothesising after the results are known)
                            • A plausible mechanism written after the result, with references that may not exist
                            • Test: state the mechanism and its falsifier before the test; resolve every DOI (digital object identifier)
                          • Timing and agents that run code
                            • Date-only timestamps
                            • Silent changes of data, windows or seeds
                            • Test: first tradable return on \(d+2\); read-only data, logged tool calls, the diff of every script before a result is accepted

                          Mini-Case: Refinements Suggested by an Assistant

                          Slide 64 in the PDF

                          A typical assistant proposal reads: "the Qwen signal works; refine it with a stronger threshold or a later entry day". Instead of testing the proposed refinement alone, the whole grid of variants is tested on the saved scores. The grid crosses 3 scores (LM, FinBERT, Qwen) with 3 thresholds, \(|s_{i,d}| > b\) for \(b \in \{0, 0.25, 0.5\}\) on the raw score, and 4 return days, \(d+2\) to \(d+5\), each a one-day position opened at the previous close: 36 strategies in all. The base case is the strategy of the previous section, 5.64 bp per day with \(t\) = 2.26.

                          Five of the 36 strategies have \(|t| > 1.96\), against 1.8 expected under the global null at 5%, and 2 of them have the wrong sign. After correction no strategy survives: 0 with Holm at an FWER of 5%, 0 with BH at an FDR of 10% and 0 with \(|t| > 3\). The Bonferroni threshold for 36 tests is \(|t| > 3.20\).

                          The refinements therefore add no evidence. The best variant is the one we started from, and it was itself chosen among three scores, on a period the model may remember.

                          In brief

                          • A typical assistant proposal:
                            • “the Qwen signal works; refine it with a stronger threshold or a later entry day”
                            • We test the whole grid on the saved scores
                          • Grid: 3 scores (LM, FinBERT, Qwen) \(\times\) 3 thresholds \(\times\) 4 return days = 36 strategies
                            • Thresholds \(|s_{i,d}| > \delta\), \(\delta \in \{0, 0.25, 0.5\}\), on the raw score
                            • Return days \(d+2\) to \(d+5\), each a one-day position opened at the previous close
                            • Base case: 5.64 bp per day, \(t\) = 2.26
                          • \(|t| > 1.96\) for 5 of 36 strategies, against 1.8 expected under the global null at 5%
                            • 2 of them have the wrong sign
                          • Survivors: Holm (FWER 5%) 0, BH (FDR 10%) 0, \(|t| > 3\) 0
                            • The Bonferroni threshold is \(|t| > 3.20\)
                          • Reading: the refinements add no evidence
                            • The best variant is the one we started from
                            • Itself chosen among three scores, on a period the model may remember
                          MFM_ch15_ai_discovery

                          The Specification Curve: Is It Different from Noise? (1/2)

                          Slide 65 in the PDF

                          A specification curve (Simonsohn, Simmons & Nelson, 2020) reports all the variants of an analysis instead of the best one. The chart sorts the 36 strategies of the mini-case by their Newey–West \(t\) statistic for the mean daily return: the horizontal axis is the rank of the specification, the vertical axis its \(t\), the colour its score (orange LM, blue FinBERT, red Qwen2.5-7B), and the dashed lines mark \(\pm 1.96\).

                          In the interactive version, the first slider sets the critical value of \(t\) and the second the number of draws under the null, from 100 to 2,000. The second panel shows the null distribution of the number of specifications above the critical value; the ▶ button raises the number of draws.

                            Question for the room Three of the 36 specifications have t > 1.96. With 2,000 draws, does this count reject the joint null at the 5% level?
                            Answer

                            No. The relevant comparison uses the joint distribution of the number of exceedances, which keeps the dependence across specifications: the signs of the positions are flipped at random in blocks of 10 days. The probability of at least 3 specifications with t > 1.96 is 0.113 in the lecture and 0.103 in the browser, both above 0.05. We do not reject the null, which does not establish that every effect is zero. No specification passes the Bonferroni cut-off, |t| > 3.20.

                            Three of the 36 specifications have \(t > 1.96\). The relevant comparison uses the joint distribution of the number of exceedances, which keeps the dependence across specifications. Under the null, at least 3 specifications with \(t > 1.96\) occur with probability 0.113, above 0.05: we do not reject the joint null, which does not establish that every effect is zero. No specification passes the Bonferroni cut-off, \(|t| > 3.20\).

                            The Specification Curve: Is It Different from Noise? (2/2)

                            Slide 66 in the PDF

                            The joint null hypothesis is that no specification predicts returns. Its distribution is simulated by flipping the signs of the positions at random in blocks of 10 consecutive trading days, zero-return days included, 2,000 times, with the same flips for all 36 specifications; this is a dependent wild bootstrap (Shao, 2010). The procedure is valid if, under the null, the signed returns are symmetric around zero and the blocks are roughly independent.

                            The observed curve has 3 specifications with \(t > 1.96\) and a median \(t\) of 0.50. Under the null, 3 or more exceedances occur with probability 0.113, and a median this high with probability 0.207. The curve is compatible with noise at 5%; reporting only the best point would hide this.

                            The result supports association, and none of the 36 predictive associations stands out from noise.

                            In brief

                            • All 36 strategies of the mini-case sorted by \(t\) (Simonsohn, Simmons & Nelson (2020)); dashed: \(\pm 1.96\)
                            • Joint null: no specification predicts returns
                              • Flip the signs of the positions at random in blocks of 10 consecutive trading days, zero-return days included, 2 000 times, with the same flips for all 36 specifications (dependent wild bootstrap, Shao (2010))
                              • Valid if, under the null, signed returns are symmetric around 0 and blocks are roughly independent
                            • Observed: 3 specifications with \(t > 1.96\), median \(t\) = 0.50
                              • Under the null, 3 or more occur with probability 0.113; a median this high with probability 0.207
                            • Reading: the curve is compatible with noise at 5%
                              • Reporting only the best point would hide this
                            • Type of evidence: association
                              • None of the 36 predictive associations stands out from noise
                            MFM_ch15_valid_inference

                            Your Turn: a Project Seed

                            Slide 67 in the PDF

                            The project question is whether LLM headline sentiment predicts returns beyond \(d+1\) on news published after the model's release, under a pre-registered grid. The first step is to replicate the Qwen strategy of this chapter, then the post-cutoff design of Lopez-Lira & Tang (2026).

                            The deliverables are the pre-registered grid and its correction, dated headlines published after the release of an open model, and the test on all variants. They also include a mechanism proposed by an LLM, stated before the test together with its falsifier, and a referee-style critique with the author's reply.

                            Each prompt that shaped a decision is logged in `AI_USE.md`, and each invented reference, memorised fact or wrong number that was caught is logged in `AI_ERRORS.md`, together with the test that caught it; templates are linked on the slide.

                            In brief

                            • Question: does LLM headline sentiment predict returns beyond \(d+1\)?
                              • On news published after the model's release
                              • Under a pre-registered grid
                            • Replicate first the Qwen strategy of this chapter, then the post-cutoff design of Lopez-Lira & Tang (2026)
                            • Deliverables
                              • The pre-registered grid and correction; dated headlines after the release of an open model; the test on all variants
                              • An LLM-proposed mechanism, stated before the test with its falsifier; a referee-style critique and your reply
                            • Log each prompt that shaped a decision in AI_USE.md
                              • Log each invented reference, memorised fact or wrong number caught in AI_ERRORS.md, with the test that caught it (templates)

                            Section summary

                            When an assistant proposes refinements, the unit of inference is the whole family of variants. On the chapter's data, 5 of 36 strategies pass \(|t| > 1.96\), none survives Holm, BH or \(|t| > 3\), and the specification curve is compatible with noise at 5%. The checks for memorisation, forking paths, HARKing and timing remain the researcher's.

                            Conclusions

                            The chapter compared four ways of measuring the sentiment of financial text on the same labelled data, tested what an LLM remembers about market history, and asked whether the measured tone of dated headlines can be traded. This section collects the main results and turns them into a practical decision.

                            Key Takeaways

                            Slide 68 in the PDF

                            Generic dictionaries misread financial text, while the LM dictionary remains a transparent baseline. FinBERT and open LLMs of 7B–14B parameters show no significant difference in accuracy on financial headlines, which is not proof of equality, and a few hundred labels with a simple classifier compete with both.

                            LLMs remember market history, so they should be evaluated only on data after their release date. Headline sentiment explains same-day price moves, but with realistic timing and costs it does not beat the market in the 16 large stocks studied.

                            Finally, scores are measurements with error. Attenuation, possibly shared errors across models, the small number of clusters and the many variants tried all shape the inference drawn from them.

                            In brief

                            • Generic dictionaries misread finance
                              • The LM dictionary is a transparent baseline
                            • FinBERT and 7B–14B open LLMs show no significant accuracy difference on financial headlines
                              • Not proof of equality
                              • A few hundred labels and a simple classifier compete with both
                            • LLMs remember market history
                              • Evaluate them only after their release date
                            • Headline sentiment explains same-day moves
                              • With realistic timing and costs it does not beat the market in the 16 large stocks studied
                            • Scores are measurements with error
                              • Attenuation, possibly shared errors across models, few clusters and many tries all shape the inference

                            Check your understanding

                            Slide 69 in the PDF

                            The four questions return to the chapter's main points:

                            1. Why word lists misclassify phrases whose sign depends on context;
                            2. Why classification accuracy does not establish trading profit;
                            3. Why the release date of a model matters for a historical backtest;
                            4. Why a sentiment score is a measurement rather than an observed truth.

                            Short model answers are in the Appendix.

                            In brief

                            • Why can a method that adds up the dictionary labels of isolated words misclassify a financial phrase?
                              • Give an example in which context changes the sign.
                            • Can a high classification accuracy establish trading profitability?
                              • Separate label quality, timing and transaction costs.
                            • Why must a historical news backtest consider the model's release date?
                              • Describe the data choice that rules out the evaluated returns being memorised in the weights.
                            • Why is a sentiment score a measurement rather than an observed truth?
                              • Explain what a random sample of human-labelled headlines allows you to estimate.
                            • Short model answers: Appendix.

                            The Practical Decision

                            Slide 70 in the PDF

                            The decision has three parts: what to choose, under which conditions, and what result would make one drop it. A fast, transparent baseline is the LM dictionary, which needs no training and can be audited. For short English financial headlines, FinBERT, or, with a few hundred labels from the domain, embeddings with logistic regression. Without labels, in other languages or for long documents, an open LLM of at least 7B parameters used zero-shot. For daily trading of the 16 large stocks studied, no sentiment signal.

                            The choices hold under out-of-sample evaluation, with intervals and paired tests. On Twitter, TF-IDF with logistic regression reaches 83.2% accuracy, against 74.2% for Qwen2.5-7B and 72.5% for FinBERT. The Qwen2.5-7B strategy breaks even at 1.5 bp per trade, below the assumed cost of 5 bp.

                            The choice should be revised after a significant McNemar test in favour of another method on texts from the target domain, after a memory test with an AUC above 0.5 over the backtest period (0.69 for Qwen2.5-14B on the months before its release), or if a signal stays significant after Holm on headlines published after the model's release, with a break-even cost above the real cost.

                            In brief

                            • What I choose
                              • A fast, transparent baseline: the LM dictionary, no training and auditable
                              • Short English financial headlines: FinBERT or, with a few hundred labels from the domain, embeddings + logistic regression
                              • No labels, other languages or long documents: an open LLM of at least 7B, zero-shot
                              • For daily trading of the 16 large stocks: no sentiment signal
                            • Under which conditions
                              • Out-of-sample evaluation, with intervals and paired tests
                              • On Twitter, TF-IDF + logistic regression reaches 83.2% accuracy, against 74.2% for Qwen2.5-7B and 72.5% for FinBERT
                              • The Qwen2.5-7B strategy breaks even at 1.5 bp per trade, below the assumed cost of 5 bp
                            • What result would make me drop it
                              • A significant McNemar test in favour of another method, on texts from the target domain
                              • A memory test with an AUC above 0.5 over the backtest period (observed here: 0.69 for Qwen2.5-14B on the months before release)
                              • A signal that stays significant after Holm on headlines published after the model's release, with a break-even cost above the real cost

                            Section summary

                            The chapter's answer to its question is negative for the data studied: models read the tone of headlines reasonably well, and that tone explains the same-day move, but on 16 large stocks with date-only timestamps it does not predict a tradable return after costs. The measurement tools of the chapter, paired tests, memory tests, attenuation corrections, PPI and corrections for multiple testing, apply to any text-based signal.

                            Further Topics

                            The frames in this part extend topics treated briefly in the body of the chapter: worked examples, human disagreement and Cohen's \(\kappa\), the structure of the errors, the prompt as a parameter, the calibration of probabilities, what Qwen2.5-14B remembers month by month, the coverage of the headlines, an event study, stability over time, the limits of the attenuation correction, PPI for regressions and the details of the GPT-4 case study.

                            Worked Example: One Sentence, Two Dictionaries

                            Slide 71 in the PDF

                            The Financial PhraseBank sentence "Uponor improved its performance in spite of the decrease in residential construction in the US", labelled positive by all annotators, shows the vocabulary problem of (15.2). LM finds one positive word, "improved", and no negative word, so \(\tau = +1.00\), correctly positive. GI finds no positive word, since "improved" is not in GI, and two negative words, "spite" and "decrease", so \(\tau = -1.00\), wrongly negative; these words describe the market, not the firm.

                            In brief

                            • Financial PhraseBank sentence, labelled positive by all annotators:
                              • Uponor improved its performance in spite of the decrease in residential construction in the US .
                            • LM: positive words improved; negative words –
                              • \(\tau = (1 - 0) / (1 + 0) = \) +1.00: positive, correct
                            • GI: positive words –; negative words spite, decrease
                              • \(\tau = (0 - 2) / (0 + 2) = \) \(-\)1.00: negative, wrong
                              • “Spite” and “decrease” describe the market, not the firm, and “improved” is not in GI

                            Annotator Agreement and Model Accuracy

                            Slide 72 in the PDF

                            The chart splits Financial PhraseBank by the highest agreement threshold each sentence reaches and shows the accuracy of LM, FinBERT, Qwen 7B and Qwen 14B in each group. On sentences with full agreement, FinBERT reaches 97%, Qwen2.5-14B 94% and LM 65%; on sentences with agreement from 50% to below 66%, the values are 70%, 55% and 54%. The human label is itself noisy and caps measured accuracy: symmetric noise independent of the classifiers shrinks measured differences towards zero, while class-dependent noise can enlarge or reverse them. Latent-label models are in the Appendix.

                            In brief

                            Annotator Agreement and Model Accuracy
                            • Financial PhraseBank split by the highest agreement threshold each sentence reaches
                              • Lowest group: 50% to below 66%
                            • Full agreement: FinBERT 97%, Qwen2.5-14B 94%, LM 65%
                              • Agreement from 50% to below 66%: FinBERT 70%, Qwen2.5-14B 55%, LM 54%
                            • The human label is itself noisy: it caps measured accuracy
                              • Latent-label models: Appendix
                              • Symmetric noise, independent of the classifiers, shrinks measured differences towards 0; class-dependent noise can enlarge or reverse them
                            • For signals: a strong, unambiguous tone is more informative than a borderline one
                            MFM_ch15_classification

                            Agreement beyond Chance: Cohen's \(\kappa\)

                            Slide 73 in the PDF

                            Accuracy counts agreement reached by chance, which is large when one class dominates. Cohen's \(\kappa = (p_o - p_e)/(1 - p_e)\) (Cohen, 1960) subtracts the chance agreement \(p_e = \sum_k p^A_k p^B_k\), computed from the marginal label frequencies of the two raters, from the observed agreement \(p_o\). On the Twitter validation set, \(\kappa\) is 0.19 for LM, 0.50 for FinBERT and 0.56 for Qwen2.5-7B, while the "always neutral" benchmark has \(\kappa = 0\), although its accuracy beats that of LM. For more than two raters per text, Fleiss' \(\kappa\) is used.

                            In brief

                            • Accuracy also counts agreement reached by chance, which is large when one class dominates
                            • Cohen's \(\kappa\) (Cohen (1960)): \(\kappa = \dfrac{p_o - p_e}{1 - p_e}\)
                              • \(p_o\): the share of texts on which two raters (here the model and the human) give the same label
                              • \(p_e = \sum_k p^{A}_k\, p^{B}_k\): the agreement expected by chance, from the marginal label frequencies of the two raters
                              • \(\kappa = 1\): perfect agreement; \(\kappa = 0\): no better than chance
                            • Twitter validation set: \(\kappa\) = 0.19 (LM), 0.50 (FinBERT), 0.56 (Qwen2.5-7B)
                            • The “always neutral” benchmark has \(\kappa = 0\), although its accuracy beats that of LM
                            • For more raters per text: Fleiss' \(\kappa\) (Appendix)

                            Where the Errors Are: Different Models, Different Mistakes

                            Slide 74 in the PDF

                            The confusion matrices on the Twitter validation set, with true labels in rows and predicted labels in columns, show three different error patterns:

                            • LM sends most positive headlines to neutral, because its positive list is short;
                            • FinBERT errs in both directions around neutral;
                            • Qwen2.5-7B has recall of about 87% on positive headlines (414 of 475) and 88% on negative ones (304 of 347), but precision of only 56% on positive (414 of 743) and 61% on negative (304 of 496), because it calls many neutral headlines positive or negative.

                            For trading, false positives cost money, so precision on the extreme classes matters.

                            In brief

                            Where the Errors Are: Different Models, Different Mistakes
                            • Twitter validation set: rows are the true labels, columns the predicted labels
                              • Counts in the cells
                            • LM sends most positive headlines to neutral: its positive list is short
                            • FinBERT errs in both directions around neutral
                            • Qwen2.5-7B: recall of about 87% on positive (414/475) and 88% on negative headlines (304/347)
                              • But precision of only 56% on positive (414/743) and 61% on negative (304/496)
                              • Many neutral headlines are called positive or negative
                            • For trading, false positives cost money: precision on the extreme classes matters
                            MFM_ch15_classification

                            Same Model, Different Wording: the Prompt Is a Parameter

                            Slide 75 in the PDF

                            Three prompts that ask the same question give different results on the Twitter validation set: accuracy ranges from 52.7% to 61.7% for 1.5B, from 73.7% to 77.8% for 7B and from 69.9% to 75.6% for 14B. The three prompts give the same label on only 67% of headlines for 7B, and its share of "neutral" answers ranges from 48% (P1) to 69% (P3). Choosing the prompt after seeing test results is a form of data snooping (Chapter 13); the prompt is fixed in advance or chosen on a separate validation set.

                            In brief

                            Same Model, Different Wording: the Prompt Is a Parameter
                            • Three prompts that ask the same question; Twitter validation set
                            • Accuracy ranges: 1.5B from 52.7% to 61.7%; 7B from 73.7% to 77.8%; 14B from 69.9% to 75.6%
                              • The three prompts give the same label on only 67% of headlines (7B)
                              • Share of “neutral” answers for 7B: 48% (P1) to 69% (P3)
                            • Choosing the prompt after seeing test results is a form of data snooping (Chapter 13)
                              • Fix the prompt in advance, or choose it on a separate validation set
                            MFM_ch15_llm_zero_shot

                            Calibration of Probabilities: ECE and Temperature Scaling

                            Slide 76 in the PDF

                            A model is calibrated if, among the texts to which it gives probability 0.9, it is right 90% of the time. The expected calibration error (ECE), \(\sum_b \frac{n_b}{n}|\text{acc}_b - \text{conf}_b|\) over 10 bins of the top probability, weighs the gap between accuracy and mean confidence in each bin by its share of texts. Temperature scaling (Guo et al., 2017) divides the logits by \(T\), as in (15.8), with \(T\) fitted by maximum likelihood on one random half of the Twitter validation set and ECE measured on the other half; it does not change the most likely class, hence not the accuracy.

                            In brief

                            • The question: when the model gives a class probability 0.9, is it right 90% of the time?
                            • ECE (expected calibration error) \(= \sum_b \frac{n_b}{n}|\text{acc}_b - \text{conf}_b|\) over 10 bins of the top probability
                              • \(n_b\) and \(n\): the number of headlines in bin \(b\) and the total number of headlines
                              • \(\text{conf}_b\): the mean top probability in bin \(b\)
                              • \(\text{acc}_b\): the share of headlines classified correctly in bin \(b\)
                            • Temperature scaling (Guo et al. (2017)): \(p_k(T) \propto p_k^{1/T}\), i.e. logits divided by \(T\)
                              • \(T\) fitted by maximum likelihood on one random half of the Twitter validation set
                              • ECE measured on the other half
                              • The saved probabilities are rounded to 6 decimals; zeros are replaced numerically by \(10^{-12}\)
                              • The most likely class does not change, hence neither does accuracy

                            Are the Probabilities Calibrated? LLMs Are Overconfident (1/3)

                            Slide 77 in the PDF

                            The reliability diagram plots, for each bin, the share of correct labels against the predicted probability of the chosen label, before temperature scaling (left) and after it (right); the dashed diagonal is perfect calibration, and the legend gives the raw ECE of FinBERT, Qwen2.5-7B and Qwen2.5-14B.

                            Are the Probabilities Calibrated? LLMs Are Overconfident (2/3)

                            Slide 78 in the PDF

                            The raw ECE is 10.4% for FinBERT, 22.6% for Qwen2.5-7B and 25.9% for Qwen2.5-14B. Qwen2.5-7B gives 88% of headlines a confidence above 0.93, with mean 99.7%, but is right on only 77% of them. Temperature scaling with \(T\) = 8.4 for 7B and 1.4 for FinBERT lowers the ECE to 4.3% and 4.6%; the fitted \(T\) for 7B depends on the numerical floor (5.6 with \(10^{-6}\) instead of \(10^{-12}\)), but the overconfidence remains. The ECE of 7B also depends on the prompt: 22.6%, 22.1% and 19.8% under P1, P2 and P3.

                            In brief

                            • Left panel: raw probabilities; right panel: after temperature scaling
                              • The diagonal means perfect calibration
                            • ECE of the raw probabilities: FinBERT 10.4%, Qwen2.5-7B 22.6%, Qwen2.5-14B 25.9%
                            • Qwen2.5-7B gives 88% of headlines a confidence above 0.93 (mean 99.7%)
                              • It is right on only 77% of them: overconfidence
                            • Temperature scaling, \(T\) = 8.4 (7B) and 1.4 (FinBERT): ECE falls to 4.3% and 4.6%
                              • \(T\) for 7B depends on the numerical floor: 5.6 with \(10^{-6}\) instead of \(10^{-12}\); the overconfidence remains
                            • ECE of 7B depends on the prompt: 22.6%, 22.1%, 19.8% under P1, P2, P3
                            MFM_ch15_valid_inference

                            Are the Probabilities Calibrated? LLMs Are Overconfident (3/3)

                            Slide 79 in the PDF

                            Two proper scores of the probabilities, both lower when better, confirm the gain: the log loss, the mean of \(-\ln p_{iy_i}\), and the Brier score (Brier, 1950), the squared distance between the probabilities and the one-hot label summed over the 3 classes. For 7B, temperature scaling moves the log loss from 2.61 to 0.63 and the Brier score from 0.468 to 0.366. With raw LLM probabilities, the net score in a regression is close to a three-valued label.

                            In brief

                            • Two proper scores of the probabilities, lower is better
                              • log loss: the mean of \(-\ln p_{i y_i}\), with \(p_{i y_i}\) the probability given to the true class \(y_i\) of headline \(i\)
                              • Brier score (Brier (1950)): squared distance between the probabilities and the one-hot label (1 for the true class, 0 otherwise), summed over the 3 classes
                            • For 7B, temperature scaling moves the log loss from 2.61 to 0.63 and the Brier score from 0.468 to 0.366
                            • With raw LLM probabilities the net score in a regression is close to a three-valued label

                            Worked Example: LLM versus FinBERT

                            Slide 80 in the PDF

                            On Twitter with prompt P1, Qwen2.5-7B reaches 74.2% and FinBERT 72.5%. With A = FinBERT and B = Qwen, \(n_{01}\) = 395 and \(n_{10}\) = 355, so the gap is 40/2,388 = +1.7 points, with 95% CI [−0.5; +3.8]; by (15.4), \(\chi^2 = 2.03\) and \(p \approx 0.15\) (exact test 0.154). Qwen2.5-14B against FinBERT gives −0.1 points, \(p\) = 0.942. The difference is not significant, which is not proof of equal accuracy: 70 to 130 times more parameters and slower processing bring no demonstrated gain on headlines, while LLMs help where labels are missing, in other languages and on long documents (Appendix).

                            In brief

                            • Twitter, prompt P1: Qwen2.5-7B 74.2%, FinBERT 72.5%
                              • A = FinBERT, B = Qwen; texts that only one model gets right: \(n_{01}\) = 395, \(n_{10}\) = 355
                              • Gap \((n_{01} - n_{10})/n\) = 40/2 388 = +1.7 points, CI 95% [\(-\)0.5; +3.8]; by hand, \(\chi^2 = (|n_{01} - n_{10}| - 1)^2/(n_{01} + n_{10})\) = 2.03, so \(p = P(\chi^2_1 > 2.03) \approx\) 0.15 (exact test: 0.154)
                              • Qwen2.5-14B against FinBERT: \(-\)0.1 points, \(p\) = 0.942
                            • No significant difference, which is not proof of equal accuracy
                              • 70 to 130 times more parameters and slower processing bring no demonstrated gain
                              • Where LLMs help: no labels needed, any language, longer and more complex documents (Appendix)

                            Memorisation Test for Qwen2.5-14B on the S&P 500 (1/2)

                            Slide 81 in the PDF

                            The chart plots, for each month from 2000, the probability that Qwen2.5-14B answers "rise" when asked only for the month, coloured by what the S&P 500 actually did that month; the dashed vertical line marks the release of the weights.

                            Memorisation Test for Qwen2.5-14B on the S&P 500 (2/2)

                            Slide 82 in the PDF

                            The model answers "fall" for 2000–2002 and 2008–2009 and "rise" for most of 2003–2007 and 2013–2021. October 2008 (−16.9%), March 2020 (−12.5%) and December 2018 (−9.2%) get \(P(\text{rise}) \approx 0\), but September 2002 (−11.0%) and February 2009 (−11.0%) get \(P(\text{rise})\) close to 1: the knowledge is coarse, made of memorable episodes rather than of every month. After the release the answers drift towards "rise", reflecting only the general belief that markets go up.

                            In brief

                            • Probability of the answer “rise” for each month, no context given
                              • Colour: what the index actually did
                            • “Fall” for 2000–2002 and 2008–2009, “rise” for most of 2003–2007 and 2013–2021
                              • October 2008 (\(-\)16.9%), March 2020 (\(-\)12.5%), December 2018 (\(-\)9.2%): \(P(\text{rise}) \approx 0\)
                            • But September 2002 (\(-\)11.0%) and February 2009 (\(-\)11.0%) get \(P(\text{rise})\) close to 1
                              • Knowledge is coarse: memorable episodes rather than every month
                            • After the release the answers drift towards “rise”
                              • No knowledge, only the general belief that markets go up
                            MFM_ch15_lookahead

                            How Many Headlines, and When: Coverage Is Uneven

                            Slide 83 in the PDF

                            The number of headlines about the 16 stocks, by the year of the trading day they are assigned to, grows strongly over time: 34,380 headlines in 2023 alone, against 8,445 for all years before 2016. Results are therefore dominated by recent years. On a news day a stock has on average 5.8 headlines, and the daily score is their average. The 16 stocks belong to today's large firms, selected with hindsight, which introduces survivorship bias.

                            In brief

                            How Many Headlines, and When: Coverage Is Uneven
                            • Headlines about the 16 stocks per year, by the trading day they are assigned to
                            • 34 380 headlines in 2023 alone, against 8 445 for all years before 2016
                              • Results are dominated by recent years; early years have few news days per stock
                            • On a news day a stock has on average 5.8 headlines
                              • Daily score: the average of the headline scores for that stock and day
                            • Survivorship bias: the 16 stocks belong to today's large firms, selected with hindsight
                            MFM_ch15_news_data

                            Event Study: the Move Happens Before and On the News (1/2)

                            Slide 84 in the PDF

                            The chart shows the mean cumulative excess return over SPY from day −5 to day +5 around the headline date, for the most positive and most negative thirds of news days by daily FinBERT score; the shaded window covers the tradable days \(d+2\) to \(d+5\). The most positive third gains +0.63% over the 5 days before the news and +0.50% on day \(d\); the most negative third gains +0.06% before and loses −0.38% on day \(d\). The move before the news points to leaks or to headlines written after the price moved.

                            In brief

                            Event Study: the Move Happens Before and On the News (1/2)
                            • Mean cumulative excess return, days \(-5\) to \(+5\)
                              • News days split by terciles of the daily FinBERT score
                              • Shaded: tradable days \(d+2\) to \(d+5\)
                            • Most positive third: +0.63% over the 5 days before, +0.50% on day \(d\)
                              • Most negative third: +0.06% and \(-\)0.38%
                              • Leaks, or headlines written after the price moved
                            MFM_ch15_news_signal

                            Event Study: the Move Happens Before and On the News (2/2)

                            Slide 85 in the PDF

                            The sentiment effect, positive minus negative third, with each stock-day weighted equally and Driscoll–Kraay intervals, is +1.00% [+0.87; +1.13] on day \(d\) and +0.17% [−0.11; +0.46] over days \(d+2\) to \(d+5\); the chart, which averages within each day first, shows +0.88% on day \(d\). The test of Boehmer, Musumeci & Poulsen (1991), robust to event-induced variance, gives \(z\) = +2.62 for the positive third and +1.25 for the negative third over \(d+2\) to \(d+5\) if events are independent, with \(N\) = 6,094 events per third, but only +1.57 and +0.66 with Driscoll–Kraay standard errors. Both thirds drift up, and the selection of today's large firms may contribute.

                            In brief

                            • Sentiment effect = positive minus negative
                              • Estimand: the difference in means, each stock-day event weighted equally; Driscoll–Kraay CI (HAC over days)
                              • Day \(d\): +1.00% [+0.87; +1.13]; days \(d+2\) to \(d+5\): +0.17% [\(-\)0.11; +0.46]
                              • The chart averages within each day first, then over the days of each third: +0.88% on day \(d\)
                            • Test robust to event-induced variance (Boehmer, Musumeci & Poulsen (1991)), \(d+2\) to \(d+5\)
                              • \(z = \overline{SCAR}/(s(SCAR)/\sqrt{N})\), with \(SCAR_e = CAR_e/\hat\sigma_e\) the cumulative excess return of event \(e\) divided by its standard deviation estimated before the event; \(N\) = 6 094 events per third
                              • \(z\) = +2.62 (positive), +1.25 (negative) if events are independent; with Driscoll–Kraay standard errors: +1.57 and +0.66
                              • Both thirds drift up; the selection of today's large firms may contribute

                            Stability over Time: Weak Evidence

                            Slide 86 in the PDF

                            The mean daily return of the strategy before costs, by subperiod, is +1.4 bp in 2010–2015, −1.8 bp in 2016–2019, +24.3 bp in 2020–2021 and −1.8 bp in 2022–2023 for FinBERT. Only 2020–2021, the meme-stock period, is large, and even then it is not significant at 5%: \(t\) = +1.89 for FinBERT, and +22.2 bp with \(t\) = +1.78 for Qwen; over the full sample the net result is negative. Published predictability relies on precise timestamps and many small stocks, as in the case study, while this sample has 16 heavily followed stocks, date-only timestamps and daily frequency.

                            In brief

                            Stability over Time: Weak Evidence
                            • Mean daily return of the strategy before costs, by subperiod
                            • FinBERT: 2010–2015 +1.4 bp, 2016–2019 \(-\)1.8 bp, 2020–2021 +24.3 bp, 2022–2023 \(-\)1.8 bp
                              • Only 2020–2021 is large, yet not significant at 5%: FinBERT \(t\) = +1.89, Qwen +22.2 bp (\(t\) = +1.78)
                              • 2020–2021: the meme-stock period (GameStop episode: Appendix)
                              • After the assumed costs the point estimates stay positive, but the statistical evidence is weak
                              • Over the full sample the net result is negative
                            • Why so little: 16 heavily followed stocks, date-only timestamps, daily frequency
                              • Published predictability relies on precise timestamps and many small stocks (case study)
                            MFM_ch15_news_signal

                            When Does the Correction Overshoot? Headlines versus Stock-Days

                            Slide 87 in the PDF

                            The correction \(b/\rho\) rests on three assumptions: non-differential error, a Twitter correlation that carries over to FNSPID headlines, and the same unit of measurement. The last one fails here: \(\rho\) is measured per headline, while \(b\) uses the daily mean of on average 5.8 headlines. If headline errors are independent and share one true tone, averaging raises the reliability by the Spearman–Brown formula \(\lambda_k = k\lambda/(1 + (k-1)\lambda)\): with \(\lambda = 0.25\) and \(k = 4\), \(\lambda_k = 0.57\) and \(\rho_k = 0.76\), so a daily slope of 10 bp is corrected to 20 bp with the headline \(\rho\) but to about 13 bp with the daily one. The \(b/\rho\) values of the chapter are then upper bounds, and labelling every headline of a random sample of stock-days would measure \(\rho\) at the right level.

                            In brief

                            • The correction \(b/\rho\) rests on three assumptions
                              • Non-differential error: given the true tone, the score error does not predict returns
                              • The Twitter correlation carries over to FNSPID headlines
                              • The same unit: \(\rho\) is measured per headline, but \(b\) uses the daily score of a stock, the mean of on average 5.8 headlines
                            • If headline errors are independent and the headlines of a stock-day share one true tone:
                              • Averaging raises the correlation with the truth
                              • The daily \(\rho\) exceeds the headline \(\rho\), and \(b/\rho\) over-corrects
                            • Numerical example: one headline has reliability \(\lambda = \rho^2 = 0.25\) (\(\rho = 0.50\))
                              • The mean of \(k = 4\) such headlines has \(\lambda_k = \dfrac{k\lambda}{1 + (k-1)\lambda} = \dfrac{4(0.25)}{1 + 3(0.25)} = 0.57\) (Spearman–Brown)
                              • So \(\rho_k = 0.76\)
                              • A daily slope of 10 bp is then corrected to \(10/0.50 = 20\) bp with the headline \(\rho\), but to \(10/0.76 \approx 13\) bp with the daily one
                            • With independent headline errors our \(b/\rho\) values are upper bounds
                              • The conclusion about the LM slope depends on this
                            • Remedy: labelling every headline of a random sample of stock-days measures \(\rho\) at the right level

                            When PPI Helps, and How It Extends to Regressions

                            Slide 88 in the PDF

                            By (15.21), the PPI interval is narrower than the one from labels alone when \(\operatorname{Var}(f)/N + \operatorname{Var}(f - Y)/n < \operatorname{Var}(Y)/n\), roughly when \(\operatorname{Var}(f - Y) < \operatorname{Var}(Y)\) for \(N \gg n\). For a regression, the rectifier applies to every moment condition that contains the predicted variable, also when it is a regressor; design-based supervised learning (Egami et al., 2023) gives a doubly robust version in which labelling probabilities may depend on the text. The few hundred labelled documents are thus not only training data but the validation sample that makes downstream inference valid.

                            In brief

                            • The PPI interval is narrower than labels alone when \(\text{Var}(f)/N + \text{Var}(f - Y)/n < \text{Var}(Y)/n\)
                              • Roughly \(\text{Var}(f - Y) < \text{Var}(Y)\) when \(N \gg n\): the model must explain part of the label's variation
                            • For a regression the rectifier applies to every moment condition that contains the predicted variable
                            • Egami et al. (2023): design-based supervised learning
                              • A doubly robust version that allows labelling probabilities to depend on the text
                            • The few hundred labelled documents are not only training data
                              • They are the validation sample that makes downstream inference valid

                            How the Authors Score News Published after the Training Cutoff of GPT-4

                            Slide 89 in the PDF

                            Lopez-Lira & Tang (2026) use 159,137 firm–headline–date observations on 4,123 US common stocks listed on the NYSE (New York Stock Exchange), NASDAQ (National Association of Securities Dealers Automated Quotations) and AMEX, from October 2021 to May 2024, after the September 2021 training cutoff of the scored model, `gpt-4-0314`. Headlines are filtered for relevance, new events and near-duplicates, and 82% are overnight news, published before 9 a.m. or after 4 p.m. The model, prompted as a financial expert, answers YES, NO or UNKNOWN on the short-term effect on the stock price at temperature 0, coded as scores 1, −1 and 0.

                            In brief

                            • 159,137 firm-headline-date observations, 4,123 US common stocks
                              • Listed on the NYSE (New York Stock Exchange), NASDAQ (National Association of Securities Dealers Automated Quotations) and AMEX (American Stock Exchange) exchanges
                              • October 2021 – May 2024
                              • The sample starts after the September 2021 training cutoff of the scored model, gpt-4-0314
                              • Filters: news-vendor relevance score 100, no “stock-gain”/“stock-loss” headlines, only new events, near-duplicates of the same day removed
                              • Overnight news (before 9 a.m. or after 4 p.m.): 82% of headlines; the rest are intraday
                            • Prompt: the model acts as a financial expert
                              • It answers YES (good news), NO (bad news) or UNKNOWN on the short-term effect on the stock price
                              • Temperature 0
                              • Score \(s_{i,t}\): YES \(= 1\), UNKNOWN \(= 0\), NO \(= -1\)

                            Portfolios and Regression in Lopez-Lira and Tang

                            Slide 90 in the PDF

                            Each day, the authors go long the stocks with a positive score and short those with a negative one, with equal weights and a leg only if it holds at least two firms; for overnight news the initial reaction runs from the previous close to the open and the drift from the open to the close. The regression \(r_{i,t} = a_i + b_t + \gamma s_{i,t} + \varepsilon_{i,t}\) has firm and date fixed effects and standard errors clustered by date and by firm, so \(\gamma\) is the return per unit of score. Twelve language models, from GPT-1 to GPT-4, are scored with the same design.

                            In brief

                            • Portfolios: each day, equal weights
                              • Long the stocks with a positive score and short those with a negative one
                              • A leg only with at least two firms
                              • For overnight news, initial reaction: previous close to open; drift: open to close
                              • Hit rate: share of days with a positive portfolio return; Sharpe ratio annualised from daily portfolio returns, as for our strategy
                            • Regression: \(r_{i,t} = a_i + b_t + \gamma\, s_{i,t} + \varepsilon_{i,t}\)
                              • Firm and date fixed effects \(a_i\), \(b_t\)
                              • Standard errors clustered by date and by firm
                              • \(\gamma\): return per unit of score
                              • Twelve language models of increasing size scored with the same design, from GPT-1 to GPT-4

                            The Drift Fades as LLMs Spread

                            Slide 91 in the PDF

                            The annualised Sharpe ratio of the GPT-4 drift strategy on overnight news falls from 6.54 in the fourth quarter of 2021 to 3.68 in 2022, 2.33 in 2023 and 1.22 in January–May 2024, more than a fivefold drop; the full-sample 2.97 pools a strong start with a weak end. The decline is consistent with the erosion predicted by the authors' model and coincides with the spread of ChatGPT, though the authors also name changing market conditions, the subperiod intervals are wide, and the one for 2024 includes zero.

                            In brief

                            The Drift Fades as LLMs Spread
                            • GPT-4 long-short drift strategy on overnight news, annualised Sharpe ratio by subperiod (source: Lopez-Lira & Tang (2026))
                            • From 6.54 in 2021Q4 to 1.22 in January–May 2024, more than a fivefold drop
                              • The full-sample 2.97 pools a strong start with a weak end: not a forecast of today's performance
                            • Consistent with the erosion predicted by the authors' model, and the timing matches the spread of ChatGPT
                            • The authors also name changing market conditions
                              • Short subperiods: the 95% intervals are wide, and the one for 2024 includes 0
                              • Intraday news: the close-to-close drift is no longer significant in 2024
                            MFM_ch15_llt_case

                            Appendix

                            The appendix contains the formal tests and derivations to which the chapter refers, the background on long documents and text regressions, the GameStop episode, the reusable prompts of the discovery loop, AI agents in trading, a checklist of pitfalls, short answers to the check questions and further readings.

                            When Annotators Disagree: Agreement and Latent Labels

                            Slide 92 in the PDF

                            Fleiss' \(\kappa\) (Fleiss, 1971) extends chance-corrected agreement to many raters, but it assumes the same number of raters for every text; PhraseBank has 5–8 raters per sentence and publishes only agreement levels, so \(\kappa\) cannot be computed from the release. The Dawid–Skene model (Dawid & Skene, 1979) gives each rater a confusion matrix, treats the true class as latent and estimates both by expectation–maximisation (EM), producing a soft label for each text; models are then evaluated by a proper scoring rule such as the log loss against these soft labels.

                            In brief

                            • Fleiss' \(\kappa\) (Fleiss (1971)): \(N\) texts, \(m\) raters, \(n_{ij}\) raters giving text \(i\) label \(j\)
                              • \(\bar P = \frac{1}{N}\sum_i \sum_j \frac{n_{ij}(n_{ij} - 1)}{m(m - 1)}\), \(\bar P_e = \sum_j p_j^2\), \(\kappa = \dfrac{\bar P - \bar P_e}{1 - \bar P_e}\); \(p_j\): share of all votes for \(j\)
                              • The formula assumes the same \(m\) for every text; PhraseBank has 5–8 raters per sentence and publishes only agreement levels, not the votes: \(\kappa\) cannot be computed from the release
                            • Dawid–Skene model (Dawid & Skene (1979)): rater \(a\) has a confusion matrix
                              • \(\pi^{(a)}_{jl} = P(\text{says } l \mid \text{true } j)\)
                              • The true class is latent
                              • EM (expectation–maximisation) estimates the \(\pi^{(a)}\) and gives each text a soft label \(P(\text{true} = j \mid \text{votes})\)
                            • Evaluate models by a proper scoring rule (log loss) against soft labels
                              • Not by accuracy against the majority label
                              • The ceiling in the agreement chart is a label-noise ceiling, not a model failure

                            Worked Example: Is FinBERT Better than LM?

                            Slide 93 in the PDF

                            On the 2,388 Twitter validation headlines, LM reaches 60.5% and FinBERT 72.5%. LM is wrong and FinBERT right on \(n_{01}\) = 539 headlines, and the reverse holds on \(n_{10}\) = 252, so the exact McNemar p-value is below 0.001. The difference is +12.0 percentage points, with bootstrap 95% CI [+9.8; +14.2]: a clear win for the fine-tuned model, even outside its training data.

                            In brief

                            • Twitter validation set, 2 388 headlines: LM 60.5%, FinBERT 72.5%
                              • LM wrong and FinBERT right: \(n_{01}\) = 539; LM right and FinBERT wrong: \(n_{10}\) = 252
                              • Exact McNemar \(p\) \(<\)0.001
                            • Difference +12.0 percentage points, bootstrap CI 95% [+9.8; +14.2]
                              • Clear win for the fine-tuned model, even outside its training data

                            Processing Time of the Qwen2.5 Models

                            Slide 94 in the PDF

                            The wall-clock time per headline on a laptop graphics processor, including the prompt and taken as the median of one pass per prompt, is 4 ms for 0.5B, 12 ms for 1.5B, 20 ms for 3B, 92 ms for 7B and 105 ms for 14B. Single passes are noisy: 7B took 51 to 94 ms and 14B 88 to 112 ms, so a claim about timing needs repeated runs.

                            In brief

                            • Wall-clock time per headline on a laptop graphics processor, including the prompt
                              • Median of one pass per prompt
                              • 0.5B: 4 ms; 1.5B: 12 ms; 3B: 20 ms; 7B: 92 ms; 14B: 105 ms
                              • Single passes are noisy: 7B took 51 to 94 ms, 14B 88 to 112 ms; a timing claim needs repeated runs

                            Beyond Headlines: Long Documents

                            Slide 95 in the PDF

                            Loughran & McDonald (2011) built their lists from 10-K filings, whose tone and uncertainty predict returns and volatility around the filing date, and Huang, Wang & Yang (2023) show that FinBERT extracts information from earnings calls and analyst reports that dictionaries miss. Kim, Muhn & Nikolaev (2023) find that LLM summaries of disclosures are much shorter yet explain stock-market reactions better than the originals, and measure disclosure "bloat". Long documents need models with long context windows, or chunking and aggregation.

                            In brief

                            • Annual reports: Loughran & McDonald (2011) built their lists from 10-K filings
                              • Tone and uncertainty predict returns and volatility around filings
                            • Earnings calls and analyst reports: Huang, Wang & Yang (2023) show FinBERT extracts information that dictionaries miss
                            • Kim, Muhn & Nikolaev (2023): LLM summaries of disclosures are much shorter
                              • But they explain stock-market reactions better than the originals
                              • They measure disclosure “bloat”: long, uninformative text is linked to lower price efficiency
                            • Long documents need models with long context windows, or chunking and aggregation

                            Text Regressions with Statistical Theory

                            Slide 96 in the PDF

                            MNIR (Taddy, 2013) models word counts given the outcome, \(c_d \sim \text{MN}(q(y_d), m_d)\) with \(q_j(y) \propto e^{\alpha_j + \varphi_j^\top y}\), so that the projection \(z_d = \Phi^\top c_d/m_d\) is sufficient for \(y_d\) given \(m_d\). SESTM (Ke, Kelly & Xiu, 2019), sentiment extraction via screening and topic modelling, screens words correlated with returns, fits a two-topic model and scores new articles by penalised likelihood, with statistical guarantees. Bybee et al. (2024) measure news attention to topics in 800,000 Wall Street Journal articles, and Jegadeesh & Wu (2013) estimate word weights from 10-K reactions; for all, the rule is to estimate on data before the forecast period.

                            In brief

                            • Taddy (2013): multinomial inverse regression: model word counts given the outcome
                              • \(c_d \sim \text{MN}(q(y_d), m_d)\) (MN: the multinomial distribution)
                              • \(c_d\): counts, \(m_d = \sum_j c_{dj}\): length
                              • Log-linear model \(q_j(y) \propto e^{\alpha_j + \varphi_j^\top y}\), loadings \(\Phi = (\varphi_1, \dots, \varphi_p)^\top\): given \(m_d\) and this model, \(z_d = \Phi^\top c_d / m_d\) is sufficient for \(y_d\)
                            • Ke, Kelly & Xiu (2019) (SESTM, sentiment extraction via screening and topic modelling)
                              • Screen words correlated with returns
                              • Fit a two-topic model of positive and negative words
                              • Score new articles by penalised likelihood
                              • The paper gives statistical guarantees for the estimator; the signal is strongest on the day of and the day after the news
                            • Bybee et al. (2024): news attention to topics in 800,000 Wall Street Journal articles
                              • It tracks economic activity and forecasts aggregate stock returns
                              • Also Jegadeesh & Wu (2013) (word weights from 10-K reactions); rule for all: estimate on data before the forecast period

                            Did the Memory Vanish? A Formal Test and Its Power

                            Slide 97 in the PDF

                            The DeLong test (DeLong et al., 1988) of equal AUC before and after the release gives \(z\) = 0.73 (\(p\) = 0.47) for 14B and \(z\) = 1.01 (\(p\) = 0.31) for 7B. With 13 up and 10 down months after the release, the standard error of the AUC under the null is 0.124, and the smallest drop detectable at 5% with 80% power is \(2.80\sqrt{0.03227^2 + 0.124^2} = 0.36\), larger than the fall from 0.69 to 0.5. We do not reject \(H_0\), but this does not show that the memory survives the release; with the same share of up months, 101 post-release months would be needed for 14B and 129 for 7B.

                            In brief

                            • \(H_0: \text{AUC}_{\text{before}} = \text{AUC}_{\text{after}}\), two independent samples
                              • DeLong variance for each AUC (DeLong et al. (1988))
                              • \(z = (\widehat{\text{AUC}}_1 - \widehat{\text{AUC}}_2)/\sqrt{\hat V_1 + \hat V_2}\): 14B \(z\) = 0.73 (\(p\) = 0.47), 7B \(z\) = 1.01 (\(p\) = 0.31)
                            • Under \(H_0\) (AUC \(= 0.5\), no ties): \(SE_0 = \sqrt{\dfrac{n_1 + n_0 + 1}{12\, n_1 n_0}}\)
                              • \(n_1\), \(n_0\): the numbers of up and down months
                              • After release: \(n_1\) = 13, \(n_0\) = 10, so \(SE_0 = \sqrt{24/1\,560}\) = 0.124
                            • Smallest detectable drop (5% two-sided test, 80% power):
                              • \(2.80\sqrt{SE_{\text{before}}^2 + SE_0^2}\) = \(2.80\sqrt{0.03227^2 + 0.124^2}\) = 0.36
                              • \(2.80 = 1.96 + 0.84\), the normal quantiles for the test and for the power; \(SE_{\text{before}}\): DeLong standard error of the 296 pre-release months
                            • A fall from 0.69 to 0.5 (0.19) is below this threshold
                              • Not rejecting \(H_0\) does not show that the memory survives release
                            • Months needed: the smallest \(m\) for which the detectable drop falls to 0.19
                              • With the post-release share of up months kept at 13/23: \(m\) = 101 (7B: 129)

                            Discrete Labels: Misclassification Is Not Classical Error

                            Slide 98 in the PDF

                            For a binary "negative news" dummy, the error of the predicted label is negatively correlated with the true label, so (15.18) does not apply (Aigner, 1973). The slope converges to \(\beta\,\pi(1 - \pi)(1 - \alpha_0 - \alpha_1)/[p(1 - p)]\), where \(\alpha_0\) and \(\alpha_1\) are the false-positive and false-negative rates, \(\pi\) the true and \(p\) the predicted share of negatives. On the Twitter validation set, with \(\pi\) = 14.5%, the factor is 0.45 for FinBERT, 0.59 for Qwen2.5-7B and 0.28 for LM: the sign is kept as long as \(\alpha_0 + \alpha_1 < 1\), but the slope can shrink to a third or less.

                            In brief

                            • Binary “negative news” dummy \(s^* \in \{0, 1\}\), predicted label \(\hat s\)
                              • \(\alpha_0 = P(\hat s = 1 \mid s^* = 0)\), \(\alpha_1 = P(\hat s = 0 \mid s^* = 1)\)
                              • The error \(u = \hat s - s^*\) is negatively correlated with \(s^*\): the classical formula does not apply (Aigner (1973))
                              • \(\operatorname{plim} \hat b = \beta\,\dfrac{\pi(1 - \pi)(1 - \alpha_0 - \alpha_1)}{p(1 - p)}\), \(\pi = P(s^* = 1)\), \(p = P(\hat s = 1)\)
                            • Twitter validation set, \(\pi\) = 14.5%, from the confusion matrices
                              • FinBERT: \(\alpha_0\) = 13.3%, \(\alpha_1\) = 23.9%, factor 0.45; Qwen2.5-7B: 0.59; LM (\(\alpha_1\) = 50.4%): 0.28
                            • As long as \(\alpha_0 + \alpha_1 < 1\) the sign is kept, but a dummy slope can shrink to a third or less

                            Two Noisy Measures: Factor Model and Instruments

                            Slide 99 in the PDF

                            In the one-factor model \(z_j = \ell_j f + u_j\) with uncorrelated errors, \(\rho_{jk} = \ell_j\ell_k\), which gives the loadings 0.53 (LM), 0.85 (FinBERT) and 0.78 (Qwen) on the daily FNSPID scores. Instrumenting the Qwen score gives a day-\(d\) effect of 59 bp [53; 66] with FinBERT as instrument and 37 bp [28; 46] with LM. If both instruments were valid the two estimates would coincide; the difference of 22 bp, \(z\) = 5.4, rejects the joint restrictions, for example because FinBERT and Qwen errors are correlated through overlapping training text, and the loadings are then suspect too.

                            In brief

                            • One factor: \(z_j = \ell_j f + u_j\), \(\text{Var}(f) = 1\)
                              • \(u_j\) uncorrelated with \(f\) and with each other
                              • Then \(\rho_{jk} = \ell_j\ell_k\), so \(\ell_j^2 = \rho_{jk}\rho_{jl}/\rho_{kl}\)
                              • Daily FNSPID scores: \(\ell_j\) = 0.53 (LM), 0.85 (FinBERT), 0.78 (Qwen)
                            • IV (instrumental variables): with \(r = a + \beta f + \varepsilon\), instrument \(z_Q\) with \(z_F\)
                              • \(\hat b_{IV} = \widehat{\text{Cov}}(r, z_F)/\widehat{\text{Cov}}(z_Q, z_F) \to \beta/\ell_Q\)
                              • If \(\ell_F \neq 0\), \(f\) is uncorrelated with \(\varepsilon\), and \(u_F\) is uncorrelated with \(u_Q\) and \(\varepsilon\)
                              • Day \(d\), Qwen: 59 bp [53; 66] with FinBERT as instrument, 37 bp [28; 46] with LM (bootstrap over days)
                            • If both instruments were valid the two estimates would coincide
                              • Difference 22 bp, \(z\) = 5.4: rejected
                              • The test rejects the joint restrictions, not a named instrument: FinBERT–Qwen errors correlated through overlapping training text are one plausible cause, an LM error related to returns another; the factor loadings \(\ell_j\) are then suspect too

                            Social Media and Crowds: GameStop, 2021

                            Slide 100 in the PDF

                            Bollen, Mao & Zeng (2011) report that collective Twitter mood predicted the Dow Jones in 2008, a result sensitive to sample and method. In January 2021, retail investors coordinating on social media pushed GameStop up by more than ten times in a few weeks; attention, sentiment and trading rose together, and this episode alone cannot separate prediction from causation. For text signals, the volume of attention can matter more than tone, and crowded signals revert and create risk for short sellers (Chapter 17).

                            In brief

                            • Bollen, Mao & Zeng (2011): collective Twitter mood predicted the Dow Jones in 2008
                              • The result is sensitive to the sample and method
                            • January 2021: retail investors coordinated on social media
                              • They pushed GameStop up by more than ten times in a few weeks
                              • Attention, sentiment and trading rose together; this episode alone cannot separate prediction from causation: each needs its own test
                            • Lessons for text signals
                              • Volume of attention can matter more than tone
                              • Crowded signals revert and create risk for short sellers (Chapter 17: bubbles)
                            Social Media and Crowds: GameStop, 2021 A GameStop store, Vallejo, California, February 2021 Photo: Will Buckner (2021); CC BY 2.0; Wikimedia Commons

                            The Discovery Loop, Reusable Prompts (1/2)

                            Slide 101 in the PDF

                            The first three steps of the loop come with prompts that can be reused:

                            1. a literature search for studies that test LLM sentiment after the model's knowledge cutoff, with every identifier resolved and Lopez-Lira & Tang (2026) as the reference design;
                            2. a request for mechanisms together with the cross-sectional pattern that would falsify each;
                            3. the reproduction of the chapter's Qwen strategy from the saved daily scores, so that no model needs to be re-run.

                            In brief

                            • Literature
                              • "Find studies since 2023 that predict stock returns with LLM sentiment and test after the model's knowledge cutoff: DOI or arXiv id, cutoff date, test window, headline estimate."
                              • Resolve every identifier
                              • Semantic Scholar, Google Scholar, Elicit for the search
                              • Lopez-Lira & Tang (2026) is the reference design
                            • Hypotheses
                              • "Propose mechanisms by which headline sentiment could predict returns after d+1 and, for each, the cross-sectional pattern that would falsify it."
                              • Under-reaction predicts drift concentrated in small, less-covered stocks
                              • Memorisation predicts an edge that disappears after the release date
                            • Data and code
                              • "From the daily news scores of this chapter and the daily closes of the course, reproduce the daily Qwen strategy of this chapter (enter at the close of d+1, hedge with SPY) and its Newey-West t."
                              • The scores are saved, so no model needs to be re-run

                            The Discovery Loop, Reusable Prompts (2/2)

                            Slide 102 in the PDF

                            The last steps of the loop:

                            1. fix robustness checks in advance: the full grid of variants and the correction, a placebo with scores shuffled across stocks within the same day, subperiods and costs of 5 bp per trade;
                            2. ask for a referee report listing every way the strategy could reflect look-ahead, memorisation, selection or timestamps, each of whose proposed tests is checked by hand;
                            3. report the whole grid, the variant chosen before the results, and the model versions and release dates.

                            In brief

                            • Robustness, fixed in advance
                              • "Before any new run, write the full grid of variants you would test and the multiple-testing correction; all of them will be reported."
                              • Placebo: scores shuffled across stocks within the same day
                              • Subperiods
                              • Costs of 5 bp per trade
                            • Critique
                              • "Write a referee report: every way this strategy could reflect look-ahead, memorisation, selection or timestamps, with the test for each."
                              • Then check each proposed test yourself: a critique can also contain invented results
                            • Report: the whole grid, the variant chosen before the results, model versions and release dates

                            AI Agents in Trading: Uses and Risks

                            Slide 103 in the PDF

                            An agent is an LLM in a loop that reads inputs, decides, calls tools such as data, code, news search or order entry through an application programming interface (API), and observes the result; agents are used today for research summaries, coding and monitoring, while fully autonomous trading is still rare. A text read by the agent can carry instructions, prompt injection, for example a fake press release. The International Monetary Fund (IMF, 2024) warns that many firms with similar models may trade alike and raise the speed and correlation of moves in stress; the controls are position limits, human approval of orders, logging of every step and tests after each model update.

                            In brief

                            • An agent is an LLM in a loop: read inputs, decide, call tools, observe, repeat
                              • Tools: data, code, news search, order entry through an API, application programming interface
                              • Used today for research summaries, coding and monitoring; fully autonomous trading is still rare and experimental
                            • Manipulation: a text read by the agent can carry instructions (prompt injection)
                              • E.g. a fake press release
                            • Systemic risk (International Monetary Fund, IMF (2024)): many firms with similar models may trade alike
                              • This raises the speed and correlation of moves in stress
                              • Controls: position limits, human approval of orders, logging of every step, tests after each model update

                            Checklist of Common Errors

                            Slide 104 in the PDF

                            The checklist collects the pitfalls of the chapter:

                            • training overlap, as for FinBERT on PhraseBank;
                            • look-ahead, tested after the model's release date;
                            • date-only timestamps, which require trading no earlier than the close after the news is surely public;
                            • snooping over prompts, thresholds and models;
                            • small daily edges that vanish after a few basis points of costs;
                            • generated regressors, whose attenuation is corrected and which are validated on a random human-labelled sample.

                            In brief

                            • Training overlap: scored on its own training data, a model looks better than it is
                              • FinBERT on PhraseBank
                            • Look-ahead: an LLM may remember the market after the news
                              • Test after its release date
                            • Timing: date-only timestamps
                              • Trade no earlier than the close after the news is surely public
                            • Snooping: prompts, thresholds and models chosen after seeing the test results
                            • Costs and crowding: small daily edges vanish after a few basis points of costs
                            • Generated regressors: scores are estimates
                              • Correct attenuation
                              • Validate on a random human-labelled sample (PPI)

                            Short Answers to the Check Questions (1/2)

                            Slide 105 in the PDF

                            A method that adds up isolated-word labels applies the negative label of "lower" in "lower costs", which is usually favourable, and counts neutral nouns such as "capital" or "board" as negative in GI. Accuracy is measured against noisy human labels on other texts, while a trade needs the first safe return (\(d+2\)), must cover \(4c\) a day and must survive the correction for many tries; in the chapter, all three scores explain same-day moves, yet no strategy earns a net profit after costs.

                            In brief

                            • Dictionary: a method that adds up isolated-word labels gives each word a single label
                              • In “lower costs” it applies the negative label of “lower” without interpreting the phrase
                              • In the usual context of firm profitability, lower costs can be favourable
                              • “capital” or “board” are neutral business nouns that GI counts as negative
                            • Accuracy and profit: accuracy is measured against noisy human labels on other texts
                              • A trade needs the first safe return (\(d+2\))
                              • It must cover \(4c\) a day and must survive the correction for many tries
                              • Here all three scores explain same-day moves, yet no strategy earns a net profit after costs

                            Short Answers to the Check Questions (2/2)

                            Slide 106 in the PDF

                            News and returns after the release of the weights cannot have been memorised in them, provided the model has no tool access to future information. The no-context memory test can flag memorisation (AUC 0.69 for 14B before the release), but a non-significant post-release AUC, 0.59 on 23 months, does not establish its absence. A score \(\hat s = s^* + u\) estimates the true tone; a random human-labelled sample allows the mean bias \(f - Y\) and the reliability to be estimated, at the unit of analysis at which the sample was measured.

                            In brief

                            • Release date: before it, the evaluated returns may already be memorised in the weights
                              • News and returns after the weight release could not have been memorised in them
                              • Condition: no tool access to future information
                              • The other sources of look-ahead bias in the backtest still need to be controlled
                            • No-context memory test: it can flag memorisation
                              • Qwen2.5-14B: AUC 0.69 on the months before the release
                              • A non-significant post-release AUC (0.59 on 23 months) does not establish the absence of contamination
                            • Score as a measurement: \(\hat s = s^* + u\) estimates the true tone
                              • Even a classical, zero-mean error can attenuate the slope (in the classical model, slope on \(z\) = effect \(\times\ \rho\))
                              • A random human-labelled sample allows the mean bias \(f - Y\) and the reliability of the score to be estimated
                              • The estimates hold for the unit of analysis at which the sample was measured
                              • PPI corrects the mean score using the mean difference \(f - Y\)
                              • The \(b/\rho\) correction requires the model assumptions and a \(\rho\) measured at the same unit of aggregation as the regression
                              • A \(\rho\) measured on headlines does not automatically justify correcting a slope on daily averages

                            Text Analysis: Readings and Applications

                            Slide 107 in the PDF

                            The core readings are Tetlock (2007) on media pessimism, Loughran & McDonald (2011) on a finance-specific dictionary and Gentzkow, Kelly & Taddy (2019) on text as data in economics. Applications to crypto-assets include classification (Pele et al., 2023), Dogecoin bubbles and sentiment (Pele, 2025) and a sentiment Quantlet (Pele, 2024); the statistics behind the tests are in Franke, Härdle & Hafner (2019).

                            In brief

                            Acronyms used in this material (1/2)

                            Slide 108 in the PDF
                            • AI: Artificial Intelligence
                            • AIC: Akaike Information Criterion
                            • AMEX: American Stock Exchange
                            • API: Application Programming Interface
                            • AUC: Area Under the (ROC) Curve
                            • BART: Bidirectional and Auto-Regressive Transformer
                            • BERT: Bidirectional Encoder Representations from Transformers
                            • BH: Benjamini–Hochberg (procedure)
                            • BY-NC: Creative Commons Attribution-NonCommercial
                            • CI: Confidence Interval
                            • CLS: CLaSsification token (BERT-type models)
                            • DOI: Digital Object Identifier
                            • ECE: Expected Calibration Error
                            • EM: Expectation-Maximisation (algorithm)
                            • FDR: False Discovery Rate
                            • FNSPID: Financial News and Stock Price Integration Dataset
                            • FWER: Family-Wise Error Rate
                            • GI: (Harvard) General Inquirer, dictionary IV-4
                            • GPT: Generative Pre-trained Transformer

                            Acronyms used in this material (2/2)

                            Slide 109 in the PDF
                            • HARKing: Hypothesising After the Results are Known
                            • IMF: International Monetary Fund
                            • IV: Instrumental Variables
                            • LLM: Large Language Model
                            • LM: Loughran-McDonald (financial sentiment dictionary)
                            • LR: Logistic Regression
                            • MIT: Massachusetts Institute of Technology
                            • ML: Machine Learning
                            • MN: MultiNomial (distribution)
                            • MNIR: MultiNomial Inverse Regression
                            • NASDAQ: National Association of Securities Dealers Automated Quotations
                            • NYSE: New York Stock Exchange
                            • OLS: Ordinary Least Squares
                            • PCA: Principal Component Analysis
                            • PPI: Prediction-Powered Inference
                            • RLHF: Reinforcement Learning from Human Feedback
                            • ROC: Receiver Operating Characteristic (curve)
                            • SE: Standard Error
                            • SESTM: Sentiment Extraction via Screening and Topic Modelling
                            • TF-IDF: Term Frequency - Inverse Document Frequency
                            • US: United States
                            • UTC: Coordinated Universal Time (Temps Universel Coordonné)
                            • VADER: Valence Aware Dictionary and sEntiment Reasoner

                            Labelled texts: Financial PhraseBank (Malo et al., 2014) and Twitter Financial News (2022). Dated headlines: FNSPID (Dong, Fan and Peng, 2024), 16 US stocks, 2009–2023; EODHD daily prices. The scores of the dictionaries, FinBERT, the Qwen2.5 models, MiniLM and TF-IDF are those computed in the chapter. Case studies: Lopez-Lira and Tang (2026), Table 5; Garcia, Hu and Rohrer (2023), replicated on the FNSPID headlines.

                            References