Neural Networks and Deep Learning

Lecture 4 · Convolutional neural networks (CNN) and transfer learning

The slide of the lecture is on the left, the same figure on the right, recomputed in your browser when you move a control. The ▶ button moves the main parameter step by step. Under each figure: a readout, a question for the room and the Quantlet with the code of the chart. The first block runs a convolution live on a real satellite image; the feature maps come from the CNN trained in the lecture; the curves are its saved training runs.

The convolutional layer · Product photos: features learned by a CNN · Deeper networks and transfer learning

The convolutional layer

Hand-made filters already find roads and field edges

Slide 10: Hand-made filters already find roads and field edges
Slide 10
Options

    Code of this chart: NN_L4_conv_edges · open in Colab

    Question for the room A 3 × 3 filter slides over a 64 × 64 Sentinel-2 tile. How many weights does it have, how large is the output, and why does the Sobel filter light up the highway?
    Answer

    Nine weights and one bias, whatever the size of the image (a dense layer from 4,096 pixels to 4,096 outputs would need about 16.8 million weights). With zero padding the output keeps 64 × 64 pixels; without it, (64 − 3)/1 + 1 = 62; with stride 2, (64 + 2 − 3)/2 + 1 = 32 (integer part). The vertical Sobel filter computes a left-minus-right difference: large positive or negative values where brightness changes from left to right, as on the edges of the highway, and zero on uniform fields. Switch to the river or the residential tile: the same filter finds other edges.

    Depth, pooling and the receptive field

    Slide 13: Depth, pooling and the receptive field
    Slide 13
    Pooling

      Code of this chart: NN_L4_receptive_field · open in Colab

      Question for the room How many 3 × 3 convolutions must be stacked before one neuron "sees" a whole 64 × 64 EuroSAT tile (640 m × 640 m)?
      Answer

      Without pooling the receptive field grows by 2 pixels per layer: 13 pixels after 6 layers, 25 after 12, so about 32 layers would be needed. With 2 × 2 pooling after every two convolutions it grows geometrically: 68 pixels after 8 convolutions, more than the whole tile. Larger kernels (5 × 5, 7 × 7) grow it faster but cost 25 or 49 weights per channel pair instead of 9; stacking small kernels with pooling is the cheaper way to see far.

      Product photos: features learned by a CNN

      Feature maps at increasing depth

      Slide 23: Feature maps at increasing depth
      Slide 23
      Block

        Code of this chart: NN_L4_fmnist_featuremaps · open in Colab

        Question for the room Feed the trained CNN the shirt of the test set. Which class does it predict, and what do the block 1 and block 3 channels respond to?
        Answer

        The CNN gives "T-shirt/top" 69.7% for this shirt: the most frequent error of the network, 102 of the 1,000 test shirts go to T-shirt/top and the recall of the class is 75.1%. Block 1 channels (14 × 14) still look like the image: edges, collars, the background. Block 3 channels (7 × 7) are coarse and respond to parts of the object; their average over the map is what the classifier reads. For the other nine test images the CNN is almost certain (99.8% or more).

        Confusion matrix: errors between products of similar shape

        Slide 21: Confusion matrix: errors between products of similar shape
        Slide 21
        Show

          Code of this chart: NN_L4_fmnist_confusion · open in Colab

          Question for the room Which product class is hardest, and does the CNN fix the errors of the softmax regression or only reduce them?
          Answer

          Shirt is the hardest class for all three models: recall 56.9% for the softmax regression, 68.2% for the MLP (multilayer perceptron) and 75.1% for the CNN; the largest confusion is always shirt → T-shirt/top (138, 124 and 102 images). Accuracy rises from 84.6% (7,850 parameters) to 89.0% (235,146) and 92.9% (140,778): the CNN has fewer parameters than the MLP and makes 37% fewer errors, because it shares weights across positions. The remaining errors are between visually similar classes (shirt, T-shirt, pullover, coat).

          Deeper networks and transfer learning

          Effects of data augmentation and residual connections

          Slide 27: Effects of data augmentation and residual connections
          Slide 27
          Networks
          Measure

            Code of this chart: NN_L4_cifar_curves · open in Colab

            Question for the room A plain network of 32 layers and a ResNet of 32 layers have the same 466,906 parameters. Why does the plain one train worse?
            Answer

            This is the degradation problem: after 12 epochs the plain network still has a training loss of 0.557 and 80.0% test accuracy, while the residual one reaches 0.291 and 87.8%. The deeper plain network is not overfitting (its training accuracy is low too); it is hard to optimise. Skip connections let each block learn a correction to the identity and let the gradient flow back unchanged. The small CNN shows the other problem: without augmentation the training accuracy reaches 96.1% against 87.2% on validation; with augmentation the gap falls to 0.9 points.

            Fine-tuning is the most accurate, and transfer matters most when labels are few

            Slide 33: Fine-tuning is the most accurate, and transfer matters most when labels are few
            Slide 33
            Models

              Code of this chart: NN_L4_transfer_curves · open in Colab

              Question for the room With only 10 labelled satellite tiles per class (100 in all), how far can each approach go?
              Answer

              A CNN trained from scratch reaches 69.9% test accuracy (mean of 2 seeds), a frozen ResNet-18 with a logistic regression on its features 78.6%, and a fine-tuned ResNet-18 87.8%. The gap closes as labels grow: with 1,000 tiles per class the three reach 94.7%, 93.0% and 98.1%, and with the full 16,200 training tiles 97.8%, 93.5% and 98.8%. Features learned on ImageNet photographs carry over to satellite images; they matter most when labels are scarce. The dip of the scratch CNN at 250 tiles (86.6% after 89.5% at 100) shows the noise of two seeds.

              Data: EuroSAT (Sentinel-2 satellite tiles of 64 × 64 pixels, 10 m per pixel, 10 land-use classes; Helber et al., 2019), Fashion-MNIST (Zalando, 28 × 28 grey product photos; 10,000 test images), CIFAR-10 (32 × 32 colour photographs). Networks and seeds (20261108) as in the lecture (analysis_L4.py); ResNet-18 = residual network with 18 layers, pre-trained on ImageNet; MLP = multilayer perceptron; ReLU = rectified linear unit.