You or the language model? When the annotators disagree
You read 10 real sentences from financial news and label each one negative, neutral or positive for an investor in the company. You play against a large language model (LLM) used as a classifier with a prompt, as in Lecture 12. Some sentences were labelled the same way by all annotators of the data set; on others the annotators were split. After each answer you see which case it was.
The model is Qwen2.5-3B-Instruct, an open LLM with 3.09 billion parameters run on a laptop, asked without examples (zero-shot): “Classify the sentiment of this sentence from financial news, from the point of view of an investor in the company. Answer with one word: positive, negative or neutral.” Its answer is the label with the highest probability for the next word; the card also shows that probability. The right answer is the label chosen by the majority of the 5–8 annotators.
What the game shows
- The model is right on 93.1% of the sentences on which all annotators agree, on 77.2% at 75% agreement, 65.7% at 66% and 53.7% when only half of them agree. A model cannot be more certain than the people who labelled the data.
- On the unanimous test sentences its probabilities are close to calibrated: the expected calibration error (ECE) is 4.7%, against 5.4% for a TF-IDF (term frequency–inverse document frequency) classifier trained on 1,581 sentences.
- The prompt is a hyper-parameter: five wordings of the same instruction move the macro-F1 of this model by 3.6 points on these sentences and by 12.9 points on finance tweets.
Data: Financial PhraseBank (Malo et al., 2014; 4,838 sentences, 5–8 annotators each). Unanimous cards: test set of the lecture (678 sentences); split cards: the 2,579 sentences with 50–75% agreement, never used for training or as examples. The game pool has 10 sentences per agreement level (at most 40 words), drawn at random (seed 20261108); every game draws 3, 3, 2 and 2 of them. Model outputs: cached outputs of analysis_L12.py. Code: lectures/L12/make_game_L12.py.

