You or the network? Sentiment of financial news
You read 10 real sentences from financial news about listed companies and decide whether each one is good news, bad news or neither for an investor in the company. After each answer you see the label given by the network of Lecture 6 and the label given by the annotators of the data set (people with a background in financial markets).
The network is the embedding bag of the lecture: it averages the vectors of the words and word pairs of a sentence, starting from the word2vec vectors learned on speeches of the European Central Bank (ECB), and passes the average to a softmax layer. It was trained on 3,385 other sentences and never saw these ones; on the 727 test sentences of this split it is right in 72.4% of cases. In the data most sentences are neutral (59.3%); the game shows more positive and negative ones.
What the game shows
- On the five test sets of the lecture the embedding bag is right in 75.0% of sentences (macro-F1 69.9%); term frequency–inverse document frequency (TF-IDF) with logistic regression reaches 77.7% (macro-F1 73.5%). With 3,385 labelled sentences, counting words still beats averaging word vectors.
- Starting from word2vec vectors learned on ECB speeches adds only 0.37 points of macro-F1: those vectors cover 84.7% of the words of the news sentences, and the language of a central bank is not the language of company news.
- Most errors move a positive or negative sentence to neutral (recall 65.9% for negative and 69.5% for positive sentences with TF-IDF); positive and negative are rarely swapped. A vector per word cannot tell what a number or a comparison means in context; this is the subject of Lecture 7.
Data: Financial PhraseBank (Malo et al., 2014; 4,838 sentences, 5–8 annotators each). The 40 sentences of the game are drawn at random from the test sentences of the first of the five random splits of the lecture (12 negative, 12 positive, 16 neutral, at most 40 words; seed 20261108); every game draws 10 of them. Predictions: cached outputs of analysis_L6.py. Code: lectures/L6/make_game_L6.py.

