Stop early or train longer? A game on training curves
You train a neural network and must decide when to stop. Each chart is a real run of Lecture 3: the training loss and, on most cards, the loss on a validation set the network does not train on. Pick one of three epochs. The right answer is the epoch whose network makes the smallest error on new data: a test set nobody looked at during training. Nine charts, about four minutes.
Seven charts come from German credit applications (600 for training, 200 for validation, 200 for test); two from card payments split by time. The loss is the cross-entropy: the lower, the better the predicted probabilities. On the first three charts you see only the training loss, as in a report that shows the fit to the training data. The model you play against is early stopping: it picks the epoch with the lowest validation loss.
What the game shows
- On German credit the training loss of the network without regularisation falls to 0.0001 after 150 epochs, while its validation loss rises from 0.536 at epoch 3 to 2.187: the network memorises its 600 applicants. A falling training loss says nothing about new clients.
- In the 9 German-credit runs of the game, with or without dropout and weight decay, the test loss is lowest between epochs 3 and 5, and the validation loss points to epochs 3 to 4.
- On card fraud, early stopping raises the test area under the precision-recall curve (PR-AUC) from 0.796 to 0.842 (5 seeds); weight decay (0.787) or dropout (0.798) alone do not.
Data: German credit (Hofmann, 1994; UCI), 1,000 applicants split 60/20/20 at random; card fraud of the ULB Machine Learning Group and Worldline (OpenML 1597), split by time. Runs: the networks of analysis_L3.py (seed 20261108), repeated with the test loss recorded after every epoch. Code: lectures/L3/make_game_L3.py.

