Neural Networks and Deep Learning

Lecture 12 · Game: You or the language model? When the annotators disagree

You or the language model? When the annotators disagree

You read 10 real sentences from financial news and label each one negative, neutral or positive for an investor in the company. You play against a large language model (LLM) used as a classifier with a prompt, as in Lecture 12. Some sentences were labelled the same way by all annotators of the data set; on others the annotators were split. After each answer you see which case it was.

The model is Qwen2.5-3B-Instruct, an open LLM with 3.09 billion parameters run on a laptop, asked without examples (zero-shot): “Classify the sentiment of this sentence from financial news, from the point of view of an investor in the company. Answer with one word: positive, negative or neutral.” Its answer is the label with the highest probability for the next word; the card also shows that probability. The right answer is the label chosen by the majority of the 5–8 annotators.

Class board for the projector

Data: Financial PhraseBank (Malo et al., 2014; 4,838 sentences, 5–8 annotators each). Unanimous cards: test set of the lecture (678 sentences); split cards: the 2,579 sentences with 50–75% agreement, never used for training or as examples. The game pool has 10 sentences per agreement level (at most 40 words), drawn at random (seed 20261108); every game draws 3, 3, 2 and 2 of them. Model outputs: cached outputs of analysis_L12.py. Code: lectures/L12/make_game_L12.py.