Neural Networks and Deep Learning

Lecture 6 · NLP foundations: text representation and word embeddings

Natural language processing (NLP) turns text into numbers. The slide of the lecture is on the left, the same idea on the right, recomputed in your browser from the course data and from the models trained in the lecture when you move a control. Under each figure: a readout, a question for the room and the Quantlet with the code of the chart.

Text as data · Sparse representations: bag of words · Dense representations: word embeddings · Sentence classifiers · Case study: topics of finance tweets

Text as data

Word frequencies and Zipf's law

Slide 7: Word frequencies and Zipf's law
Slide 7
Axis scale
Line

    Code of this chart: NN_L6_zipf · open in Colab

    Question for the room Our word2vec keeps only the words that occur at least 5 times in the ECB speeches. How many distinct words does that remove, and how much of the running text do we lose?
    Answer

    The speeches have 20,130 distinct words; a minimum count of 5 keeps 8,539 of them (42%) but still covers 98.4% of the 1,289,896 tokens. Rare words are many but each is rare, so the lost text is only 1.6%. Raise the threshold to 100: 1,545 words still cover 86.1% of the text. This is Zipf's law at work (slope −1.498 on the log–log scale): a small vocabulary covers most of the text, and the long tail is where out-of-vocabulary (OOV) words come from.

    Sparse representations: bag of words

    The linear model learns a finance sentiment dictionary

    Slide 11: The linear model learns a finance sentiment dictionary
    Slide 11
    Preprocessing

      Code of this chart: NN_L6_top_words · open in Colab

      Question for the room The first sentence, "… operating profit amounted to EUR 11.9 mn, down from EUR 15.0 mn a year earlier", is negative with probability 96.3% for the bag-of-words (BoW) logistic regression. What happens if we remove the English stop words before scoring it?
      Answer

      The model loses the word "down", which the scikit-learn stop-word list removes (together with "up", "not", "below" and "above"), and "down" is the strongest negative word of the sentence (β_positive − β_negative = −4.41). Without it the sentence is predicted neutral (41.0%, against 29.7% negative). The same happens to "Operating profit was EUR 139.7 mn, up 23% …", which falls from 83.2% positive to a negative prediction (44.8%). Generic preprocessing removes exactly the words that carry financial sentiment; type your own sentence to test other words.

      Dense representations: word embeddings

      Nearest neighbours in our word2vec and GloVe spaces

      Slide 19: Nearest neighbours in our word2vec and GloVe spaces
      Slide 19

        Code of this chart: NN_L6_w2v_pca · open in Colab

        Question for the room Which words are closest to "interest" in the ECB speeches? Does the vector of "interest" also know the sense "attention, curiosity"?
        Answer

        Both models put "rates" and "rate" first (cosine 0.83 and 0.80 in word2vec, 0.91 and 0.83 in GloVe): in central-bank speeches "interest" almost always means the interest rate. A static embedding has one vector per word, built from all its contexts, so the rarer sense disappears. Words shared by the two top lists are coloured: for "inflation" only 2 of the top 5 are common (headline, underlying); the lecture reports 1.88 shared words on average over its eight probe words.

        A map of the embedding space

        Slide 19: A map of the embedding space
        Slide 19
        Word groups
        Vectors

          Code of this chart: NN_L6_w2v_pca · open in Colab

          Question for the room Projected on two principal components (principal component analysis, PCA), do the four groups of words form separate regions? How much of the variation of the 100-dimensional vectors does the map keep?
          Answer

          Yes: monetary-policy, energy, banking and payment words occupy different regions, although the map keeps only about 32% of the variance of the 32 vectors (31.8% for word2vec and 31.5% for GloVe as computed here from vectors rounded to four decimals; the lecture reports 31.9%). Keep only monetary policy and energy: the first two components now keep 45.9% (word2vec), because the map has fewer directions to show. A 2-D map is a picture, not a measure of similarity; use the cosine of the full vectors for that.

          Sentence classifiers

          Classification errors and annotator agreement

          Slide 30: Classification errors and annotator agreement
          Slide 30
          Agreement of the annotators

            Code of this chart: NN_L6_agreement · open in Colab

            Question for the room TF-IDF (term frequency–inverse document frequency) with logistic regression is right on 77.7% of the PhraseBank test sentences. Is it equally good on the sentences on which the annotators disagreed?
            Answer

            No. On the sentences on which all annotators agreed it is right 88.4% of the time, on those with only 50% agreement 58.1% of the time, hardly better than always answering "neutral" (56.0%). The confusion matrix shows where the errors go: positive and negative sentences are mostly pushed to neutral (263 positive and 107 negative sentences over the five test sets). The other models, including the embedding bag, have the same profile: label noise caps the accuracy, so check the agreement of your own annotators before buying a bigger model.

            Case study: topics of finance tweets

            Topic classification: F1 is high for earnings and the Fed, low for general news

            Slide 34: Topic classification: F1 is high for earnings and the Fed, low for general news
            Slide 34
            Model

              Code of this chart: NN_L6_topic_f1 · open in Colab

              Question for the room The topic "Gold | Metals | Materials" has the lowest F1 score of TF-IDF + logistic regression (45.2%). Why?
              Answer

              It has only 13 test tweets, and its words overlap with macroeconomic and market news: 7 of the 13 are classified correctly, 2 go to "Macro". With so few examples one error moves the F1 by several points; the embedding bag reaches 58.3% on the same topic while being worse on average (macro-F1 80.45% against 82.62%). Choose "Dividend" to see the easiest topic (F1 96.4%): its vocabulary is specific.

              Data: speeches of the European Central Bank (ECB), 8 January 2022 to 30 October 2026, 400 speeches; Financial PhraseBank (Malo et al., 2014), 4,838 sentences of financial news, 5–8 annotators each; Twitter financial news, sentiment and topic (Hugging Face, zeroshot). The word vectors (word2vec, GloVe) and the classifiers were trained by lectures/L6/analysis_L6.py; the bag-of-words logistic regression of the second block is the model of the lecture (PhraseBank split 1), refitted here with the same settings.