Which feature drove the decision?
A bank's network has approved or refused a real home-equity loan application. For each of 8 applicants you see the file and the network's decision, and you choose, among four features, the one that pushed this decision the most. Then you see the answer of SHAP (SHapley Additive exPlanations, Kernel SHAP) and whether LIME (local interpretable model-agnostic explanations) agrees.
The network is the multilayer perceptron (MLP) of Lecture 11, trained on 3,576 other loans (area under the receiver operating characteristic curve, AUC, 0.937 on the test loans); it refuses a loan when its probability of default is at least 50%. The right answer is the option with the largest absolute Kernel SHAP contribution to the log-odds of default; LIME's answer is the option with the largest absolute LIME slope × standardised value. Both explanations were computed with the settings of the lecture.
What the game shows
- LIME and SHAP name the same top feature for 19 of the 30 applicants of the game. For the applicant studied in the lecture, SHAP puts first the missing debt-to-income ratio (DEBTINC missing) and LIME the delinquent credit lines (DELINQ); the Spearman correlation of their absolute contributions is 0.80.
- Explanations are estimates and move with chance: with 500 samples LIME keeps the same top feature in only 73% of 30 runs, and across the 10 training seeds of the same network an applicant's top SHAP feature agrees for only 78% of the pairs of seeds.
- Before an explanation reaches a customer or a supervisor, check it: two methods, several seeds, fixed and reported settings.
Data: HMEQ home-equity loans (Baesens, Roesch and Scheule, 2016; 5,960 loans). The 30 applicants of the game are drawn at random from the test applicants explained in the lecture (12 refused and 18 approved by the network; seed 20261108); every game draws 8 of them. Kernel SHAP and the seed comparison come from the lecture's cached results; LIME is recomputed with the lecture's settings (5,000 samples, default kernel width, seed 0). Code: lectures/L11/make_game_L11.py.

