You or FinBERT? Finance tweets on which the models disagree
You read 10 real finance tweets and decide whether each one is bearish (bad news for the price), neutral or bullish (good news for the price). After each answer you see what FinBERT decided, what two simple models decided, and the true label. The tweets were chosen on purpose: they are cases where the simple models and FinBERT do not agree.
FinBERT is a BERT (bidirectional encoder representations from transformers) model pretrained on financial text and fine-tuned in the lecture on 8,588 labelled tweets. The two simple models are term frequency–inverse document frequency (TF-IDF) with logistic regression and the embedding bag; neither looks at the order of the words. Of the 2,388 test tweets, in 230 the two simple models give the same label and FinBERT a different one; the game draws its cards at random from these 230. Links in the tweets are removed.
What the game shows
- On all 230 tweets of this kind FinBERT is right in 63.0% of cases, the two simple models in 31.7%. Example: “Beckhams’ business empire sees revenues drop 18%”: both simple models say bullish, FinBERT and the label say bearish. Attention lets every word take its meaning from the other words of the text.
- On all 2,388 test tweets fine-tuned FinBERT reaches a macro-F1 of 84.8%, TF-IDF with logistic regression 75.5%: a difference of 9.3 points, 95% bootstrap interval [7.3; 11.3].
- The gain comes from pretraining, not from attention alone: our small transformer trained from scratch reaches 68.8%, below TF-IDF, and FinBERT without fine-tuning 66.9%.
Data: Twitter financial news sentiment (Hugging Face, zeroshot), published validation split (2,388 tweets), the test set of the lecture. Disagreement set: TF-IDF + logistic regression and embedding bag (first run) agree, FinBERT fine-tuned (first seed) differs; 230 tweets. The 30 cards of the game are drawn at random from them (seed 20261108); every game draws 10. Predictions: cached outputs of analysis_L7.py. Code: lectures/L7/make_game_L7.py.

