Statistics of Financial Markets

Chapter 12 · Scoring Models

The slide of the lecture is on the left, the same figure or table on the right, computed in the browser from the same data. The ▶ button moves the parameter step by step; the tables and worked examples are recomputed with it. Abbreviations: PD (probability of default), LGD (loss given default), EAD (exposure at default), EL (expected loss), IRB (internal ratings-based approach), RWA (risk-weighted assets), WoE (weight of evidence), IV (information value), LDA (linear discriminant analysis), ROC (receiver operating characteristic), AUC (area under the ROC curve), CAP (cumulative accuracy profile), AR (accuracy ratio), KS (Kolmogorov–Smirnov statistic), HL (Hosmer–Lemeshow test), TP, FN, FP, TN (true positive, false negative, false positive, true negative), TPR (true positive rate), FPR (false positive rate), PDO (points to double the odds), LR (likelihood ratio), CI (confidence interval), se (standard error), DM (Deutsche Mark), EBIT (earnings before interest and taxes), TA (total assets), DD (distance to default), GBM (geometric Brownian motion).

The Credit Score: Definition · Default Data · Weight of Evidence and Information Value · Logistic Regression · Linear Discriminant Analysis · Building the Scorecard · Validation · Reject Inference and Fairness · Corporate Default: Altman and Merton

The Credit Score: Definition

PD in Good and Bad Years: the One-Factor Model

Slide 10: PD in Good and Bad Years: the One-Factor Model
Slide 10
    Question for the room ▶ raises ρ from 0.2 to 0.5. What happens to the probability that a loan with PD = 2% defaults in a bad year (y = −3)?
    Answer

    It rises from 21.3% to 53.8%; in a typical year (y = 0) it falls from 1.08% to 0.18%. A larger weight of the common factor concentrates the defaults in the bad years, while the average over all years stays at 2%.

    Worked Example: IRB Capital of a Retail Loan

    Slide 11: Worked Example: IRB Capital of a Retail Loan
    Slide 11

    Worked example: IRB capital, other retail exposures

    \[K = \mathrm{LGD}\Big[\Phi\Big(\frac{\Phi^{-1}(\mathrm{PD}) + \sqrt{R}\,\Phi^{-1}(0.999)}{\sqrt{1 - R}}\Big) - \mathrm{PD}\Big], \quad R = 0.03\,w + 0.16\,(1 - w), \quad w = \frac{1 - e^{-35\,\mathrm{PD}}}{1 - e^{-35}}\]
        Question for the room ▶ raises PD from 2% to 4%. Does the capital K double?
        Answer

        No: it rises from 4.64% to 5.20% of EAD (+12%), while EL doubles from 0.90% to 1.80%. The correlation R falls from 0.095 to 0.062 as PD rises, and the extra defaults are increasingly expected rather than unexpected.

        Expected Loss and IRB Capital as Functions of PD

        Slide 12: Expected Loss and IRB Capital as Functions of PD
        Slide 12
          Question for the room Other retail exposures, LGD 45%: PD rises from 5% to 10%, so EL doubles (from 2.25% to 4.50% of EAD). What happens to the capital K?
          Answer

          It rises only from 5.31% to 6.04% of EAD (+14%). K covers the unexpected loss: the higher the PD, the larger the share of the loss that is expected and priced in, and R falls from 0.053 to 0.034. Above a PD of about 16%, K is even smaller than EL.

          Default Data

          Bad Rate by Checking Account

          Slide 15: Bad Rate by Checking Account
          Slide 15
            Question for the room In the old Statlog labels, the 394 credits with a balance of at least 200 DM were called “no checking account” and looked like the safest group (11.7%). In the corrected data, what is the bad rate of the credits without a checking account?
            Answer

            49.3% (274 credits), the riskiest group, against 11.7% for a balance of at least 200 DM or a salary account: a ratio of 4.2. The figures were right in both versions; only the labels were swapped.

            Weight of Evidence and Information Value

            Worked Example: WoE and IV of the Checking Account

            Slide 19: Worked Example: WoE and IV of the Checking Account
            Slide 19

            WoE table on the full sample

            Worked example: WoE and IV term of one row

            \[\mathrm{WoE}_i = \ln\frac{g_i/G}{b_i/B}, \qquad \mathrm{IV} = \sum_i \Big(\frac{g_i}{G} - \frac{b_i}{B}\Big)\mathrm{WoE}_i\]
                Question for the room Group “≥ 200 DM or salary account”: 348 of the 700 goods and 46 of the 300 bads. What is its WoE?
                Answer

                ln(0.497/0.153) = 1.176: the share of goods is 3.2 times the share of bads. Its IV term, (0.497 − 0.153) × 1.176 = 0.404, is the largest of the four; the total IV is 0.666, a strong variable.

                WoE of the Credit Duration

                Slide 21: WoE of the Credit Duration
                Slide 21
                  Question for the room ▶ moves to the credit amount (bins up to 1300, 2000, 3000, 5000 DM and above). Does the bad rate rise steadily with the amount?
                  Answer

                  No: it is 30.9% for the smallest loans, falls to 23.9% at 2001–3000 DM and rises to 41.5% above 5000 DM (WoE from 0.31 to −0.50). The relation is U-shaped, which a single linear term in a logit misses and the bins capture; IV = 0.082, a weak variable.

                  Information Value of the 20 Variables

                  Slide 22: Information Value of the 20 Variables
                  Slide 22
                    Question for the room On the full sample, property has IV = 0.113 ≥ 0.1. Does it also pass the 0.1 threshold on the training sample, where the scorecard variables are chosen?
                    Answer

                    No: on the training sample its IV is 0.082, and employment duration (0.086 on the full sample, 0.122 on the training sample) takes its place. Six variables pass in both cases. IV computed on a few hundred credits moves by several hundredths between samples: on the 300 test credits the IV of the checking account is 0.913, against 0.666 on all 1,000.

                    Logistic Regression

                    Odds and Log-Odds

                    Slide 23: Odds and Log-Odds
                    Slide 23

                    Worked example: odds and log-odds

                    \[\text{odds} = \frac{p}{1 - p}, \qquad \mathrm{logit}(p) = \ln\frac{p}{1 - p}, \qquad \mathrm{logit}(1 - p) = -\mathrm{logit}(p)\]
                        Question for the room p = 0.2 gives the log-odds −1.386. ▶ moves p to 0.8. What are the log-odds there?
                        Answer

                        +1.386: logit(1 − p) = −logit(p), the logit is symmetric around p = 0.5. The odds are 4 (four defaults per repaid credit), the inverse of 0.25.

                        A Logistic Curve: PD and Duration

                        Slide 25: A Logistic Curve: PD and Duration
                        Slide 25
                          Question for the room ▶ fits the logit on age alone. By how much do ten more years multiply the odds of default?
                          Answer

                          By e^(10 × (−0.0185)) = 0.83, that is −17%: PD falls from 34.0% at age 25 to 21.3% at age 60. Duration alone gives 1.57 per year: older applicants are safer, longer credits riskier.

                          A Small Logit on the Full Sample

                          Slide 27: A Small Logit on the Full Sample
                          Slide 27

                          Coefficients, full sample (1,000 credits)

                            Question for the room ▶ drops duration from the model. What happens to the coefficient of the amount (0.029, p = 0.368)?
                            Answer

                            It rises to 0.112 per 1000 DM, p < 0.001: without duration, the amount picks up the risk of long credits, with which it is correlated. A coefficient is the effect with the other variables held fixed; leaving out a correlated variable changes it.

                            Odds Ratios with Confidence Intervals

                            Slide 28: Odds Ratios with Confidence Intervals
                            Slide 28
                              Question for the room The 95% interval for age (+10 years) is 0.74–0.98. Does the 99% interval contain 1?
                              Answer

                              Yes: 0.71–1.02. The age effect has p = 0.022: significant at 5%, not at 1%. The intervals of duration and of both account indicators exclude 1 also at 99%; the amount contains 1 at every level.

                              Worked Example: the PD of an Applicant

                              Slide 29: Worked Example: the PD of an Applicant
                              Slide 29

                              Worked example: PD of an applicant (rounded coefficients)

                              \[\eta = -2.101 + 0.385\,\tfrac{\text{dur}}{12} - 0.159\,\tfrac{\text{age}}{10} + 0.029\,\tfrac{\text{amount}}{1000} + 1.859\,\mathbb{1}_{\text{no acc.}} + 1.338\,\mathbb{1}_{<0}, \qquad \mathrm{PD} = \frac{1}{1 + e^{-\eta}}\]
                                  Question for the room ▶ raises the duration from 36 to 48 months for the same applicant. What is the PD?
                                  Answer

                                  71.8%: η rises by 0.385 to 0.937. The approximation p(1 − p)β = 8.9 points from 63.5% gives 72.4%, close, but the exact change, 8.3 points, is smaller because the logistic curve flattens as p moves away from 0.5.

                                  The WoE Logit Scorecard

                                  Slide 30: The WoE Logit Scorecard
                                  Slide 30

                                  The WoE logit scorecard (training sample, 700 credits)

                                    Question for the room ▶ lowers the IV threshold to 0.02, so 15 WoE variables enter instead of 6. Does the test AUC rise?
                                    Answer

                                    No: it falls from 0.801 to 0.787, while the training AUC rises from 0.789 to 0.825. The nine weak variables fit the noise of the 700 training credits.

                                    Correcting for Oversampling

                                    Slide 31: Correcting for Oversampling
                                    Slide 31

                                    Worked example: the prior correction

                                    \[\ln \text{odds}_{\text{pop}} = \ln \text{odds}_{\text{sample}} + \ln\frac{\pi/(1 - \pi)}{\pi_s/(1 - \pi_s)}\]
                                        Question for the room ▶ lowers the population bad rate π. For the applicant with a sample PD of 63.5%, what is the PD at π = 1%?
                                        Answer

                                        3.9%: the correction is ln(0.01/0.99) − ln(0.3/0.7) = −3.748, so η = 0.552 − 3.748 = −3.196. Only the constant moves; the ranking of the applicants, and with it the AUC, stays the same.

                                        Linear Discriminant Analysis

                                        Worked Example: Fisher's Direction with Two Variables

                                        Slide 34: Worked Example: Fisher's Direction with Two Variables
                                        Slide 34

                                        Worked example: Fisher’s direction

                                        \[J(w) = \frac{[w^\top(m_1 - m_0)]^2}{w^\top S_W w}, \qquad w = S_W^{-1}(m_1 - m_0)\]
                                            Question for the room Duration and age alone: what in-sample AUC does Fisher’s discriminant reach on the 1,000 credits?
                                            Answer

                                            0.640, exactly as much as the logit with the same two variables. The scorecard with six WoE variables reaches 0.801 on the test sample: two variables are far from enough, as the overlapping clouds also show.

                                            Fisher's Boundary and the Logit Boundary

                                            Slide 35: Fisher's Boundary and the Logit Boundary
                                            Slide 35
                                              Question for the room ▶ moves the PD on the boundary from 30% to 50%. What happens to the two lines?
                                              Answer

                                              They move in parallel to longer durations: at age 30 from about 20 months (19.9 LDA, 19.4 logit) to about 41 months (40.9 and 42.0). The slope does not depend on the PD level; only the constant changes.

                                              Comparison of Logit and LDA

                                              Slide 36: Comparison of Logit and LDA
                                              Slide 36

                                              Logit and LDA on the same six WoE variables

                                                Question for the room On the training sample (700 credits), is the AUC of the logit higher than that of the LDA?
                                                Answer

                                                No, slightly lower: 0.789 against 0.790, DeLong z = −0.48, p = 0.63. On the test sample the logit is slightly ahead (0.8013 against 0.8009, z = 0.18). The two scores correlate 0.999; neither ranking is separated from noise by this test.

                                                Building the Scorecard

                                                Worked Example: Scores and Points

                                                Slide 38: Worked Example: Scores and Points
                                                Slide 38

                                                Worked example: scores and points

                                                \[\text{Score} = \text{Offset} + \text{Factor}\,\ln\frac{1 - p}{p}, \quad \text{Factor} = \frac{\text{PDO}}{\ln 2}, \quad \text{Offset} = 600 - \text{Factor}\,\ln 50\]
                                                    Question for the room ▶ sets PDO to 40 points. What score does a PD of 5% get now (572.1 with PDO = 20)?
                                                    Answer

                                                    544.2: Factor 57.71, Offset 374.25. The base point stays at 600 for odds of 50:1; a PD of 5% means odds of 19:1, below the base, and with a steeper scale the score falls further below 600.

                                                    Scores of the Test Sample

                                                    Slide 39: Scores of the Test Sample
                                                    Slide 39
                                                      Question for the room The cut-off at PD = 1/6 is 534 points. How many of the 90 bad test credits score at least 534 and would be accepted?
                                                      Answer

                                                      8 (FN): 82 of the 90 bads are rejected. The price: 112 of the 210 goods are rejected. The two distributions overlap, so every cut-off trades one error for the other.

                                                      Validation

                                                      Two Cut-Offs on the Test Sample

                                                      Slide 41: Two Cut-Offs on the Test Sample
                                                      Slide 41

                                                      Confusion matrices on the test sample (last row: the chosen cut-off)

                                                      Worked example: the rates at the chosen cut-off

                                                      \[\mathrm{TPR} = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FN}}, \quad \mathrm{FPR} = \frac{\mathrm{FP}}{\mathrm{FP} + \mathrm{TN}}, \quad \text{cost} = \frac{5\,\mathrm{FN} + \mathrm{FP}}{n}\]
                                                          Question for the room ▶ lowers the cut-off from 0.5 to 0.3. Does accuracy rise or fall, and what happens to the cost?
                                                          Answer

                                                          Accuracy falls from 76.0% to 72.7%, the cost falls from 0.867 to 0.527: TPR rises from 47.8% to 78.9%, FPR from 11.9% to 30.0%. Accuracy weighs both errors equally; the cost weighs an accepted bad five times.

                                                          Expected Cost and the Choice of the Cut-Off

                                                          Slide 42: Expected Cost and the Choice of the Cut-Off
                                                          Slide 42
                                                            Question for the room ▶ raises the cost of accepting a bad from 5 to 10. Which share of the applicants does the Bayes cut-off reject?
                                                            Answer

                                                            81%: the cut-off falls from 1/6 to 1/11 = 0.091 and the share rejected rises from 64.7% to 81.0%; the expected cost is 0.620, below the 0.700 of rejecting everybody.

                                                            ROC Curve and AUC

                                                            Slide 43: ROC Curve and AUC
                                                            Slide 43

                                                            Worked example: AUC as a rank probability

                                                            \[\mathrm{AUC} = P(s_{\text{bad}} > s_{\text{good}}) + \tfrac12 P(s_{\text{bad}} = s_{\text{good}}), \qquad \text{Gini} = 2\,\mathrm{AUC} - 1\]
                                                                Question for the room The checking account alone has four values. What share of the 18,900 (bad, good) pairs are tied?
                                                                Answer

                                                                23.1% (4,362 pairs); each counts 1/2, so AUC = (11,579 + 2,181)/18,900 = 0.728. The WoE logit has only 12 tied pairs (0.06%).

                                                                ROC Curves on the Test Sample

                                                                Slide 44: ROC Curves on the Test Sample
                                                                Slide 44
                                                                  Question for the room On the training sample (700 credits), is the AUC of the WoE logit above or below 0.801, the value on the test sample?
                                                                  Answer

                                                                  Below: 0.789. In-sample performance is usually optimistic, but this split was lucky: over the 100 folds of the cross-validation the mean test AUC is 0.764.

                                                                  CAP Curve, Accuracy Ratio and KS

                                                                  Slide 45: CAP Curve, Accuracy Ratio and KS
                                                                  Slide 45

                                                                  Worked example: KS, Gini and AR

                                                                  \[\mathrm{KS} = \max_s |F_{\text{bad}}(s) - F_{\text{good}}(s)| = \max_c\,(\mathrm{TPR} - \mathrm{FPR}), \qquad \mathrm{AR} \approx \text{Gini}\]
                                                                      Question for the room The checking account alone: is its accuracy ratio AR equal to its Gini, 0.456?
                                                                      Answer

                                                                      No: AR = 0.240. With only four score values, the CAP orders applicants with the same score in file order, which is arbitrary; AR = Gini holds for a continuous score (WoE logit: 0.602 and 0.603). KS is 0.410, against 0.511.

                                                                      CAP and KS on the Test Sample

                                                                      Slide 46: CAP and KS on the Test Sample
                                                                      Slide 46
                                                                        Question for the room Rejecting the riskiest 30% of the 300 test applicants, what share of the 90 bads do we catch?
                                                                        Answer

                                                                        60% (54 of 90). The perfect model would catch all of them, since 30% is exactly the bad rate; a random score would catch 30%.

                                                                        CAP and KS on the Test Sample

                                                                        Slide 46: CAP and KS on the Test Sample
                                                                        Slide 46
                                                                          Question for the room At the Bayes cut-off, 534 points, how large is the gap F_bad(s) − F_good(s), compared with KS = 0.511 at 516 points?
                                                                          Answer

                                                                          0.373: 91.1% of the bads and 53.8% of the goods score at most 534. This is TPR − FPR at this cut-off; KS is its maximum over all cut-offs. The cut-off that minimises the expected cost is not the one that maximises KS, because KS weighs both errors equally.

                                                                          Calibration and the Brier Score

                                                                          Slide 47: Calibration and the Brier Score
                                                                          Slide 47

                                                                          Worked example: the Brier score and the Hosmer–Lemeshow test

                                                                          \[\mathrm{BS} = \frac1n\sum_i (p_i - y_i)^2, \qquad \mathrm{HL} = \sum_g \frac{(O_g - n_g\bar p_g)^2}{n_g\bar p_g(1 - \bar p_g)} \approx \chi^2(G - 2)\]

                                                                            Calibration table of the scorecard (test sample)

                                                                              Question for the room Scorecard test sample: with 5 groups of 60 credits instead of 10, what p-value does the Hosmer–Lemeshow test give?
                                                                              Answer

                                                                              0.90 (HL = 0.589, 3 degrees of freedom); 0.59 with 10 groups, 0.82 with 20. Not rejected in any grouping: with 300 credits the test has little power, so “not rejected” does not prove that the PDs are exact.

                                                                              Calibration on the Taiwan Data

                                                                              Slide 48: Calibration on the Taiwan Data
                                                                              Slide 48

                                                                              Calibration table (in red: HL term above 3.84)

                                                                                Question for the room With 5 groups of 1,800 clients instead of 10, does the Hosmer–Lemeshow test still reject calibration at 5%?
                                                                                Answer

                                                                                Yes: HL = 22.6 with 3 degrees of freedom, p < 0.001 (with 20 groups, 67.5, p < 0.001). With 9,000 clients the test detects deviations of one or two percentage points; what matters is the size of the deviations, not only the p-value.

                                                                                Training and Test AUC in Repeated Cross-Validation

                                                                                Slide 50: Training and Test AUC in Repeated Cross-Validation
                                                                                Slide 50
                                                                                  Question for the room ▶ shows the first 1, 5, 10 and 20 repetitions. How much does the mean test AUC of the WoE logit move?
                                                                                  Answer

                                                                                  Very little: 0.766, 0.767, 0.766, 0.764. A single 5-fold run already averages five test folds; the single split of the lecture (0.801) is one draw from a distribution with standard deviation 0.032.

                                                                                  Reject Inference and Fairness

                                                                                  A Reject-Inference Experiment on the Taiwan Data

                                                                                  Slide 54: A Reject-Inference Experiment on the Taiwan Data
                                                                                  Slide 54

                                                                                  The reject-inference experiment (test sample, 9,000 clients)

                                                                                    Question for the room ▶ lets the old score accept 90% instead of 70%. Does the model trained on the accepted still underestimate the risk of the rejected?
                                                                                    Answer

                                                                                    No, it now overestimates it: mean PD 80.0% against an observed 69.4%, and its AUC on all applicants (0.777) is that of the full model. At 90%, clients who paid late in September are among the accepted, so no variable drops out; the rejected 10% lie beyond the training data, and the logit extrapolates.

                                                                                    Approval and Default by Group: Taiwan

                                                                                    Slide 56: Approval and Default by Group: Taiwan
                                                                                    Slide 56
                                                                                      Question for the room ▶ raises the approval share from 75% to 90%. Does the gap between the approval rates of women and men grow or shrink?
                                                                                      Answer

                                                                                      It shrinks from 4.1 to 2.2 percentage points (90.9% against 88.7%). The order of the groups stays the same, because the model is calibrated in each group and men default more often (24.7% against 20.5%).

                                                                                      Corporate Default: Altman and Merton

                                                                                      Worked Example: a Z-Score

                                                                                      Slide 59: Worked Example: a Z-Score
                                                                                      Slide 59

                                                                                      Worked example: the Z-score

                                                                                      \[Z = 1.2X_1 + 1.4X_2 + 3.3X_3 + 0.6X_4 + 1.0X_5\]
                                                                                          Question for the room ▶ raises EBIT/TA from 0. At which value does the company leave the grey zone (Z > 2.99)?
                                                                                          Answer

                                                                                          Above 0.27 (exactly 0.2697): the other four terms give 2.100, and 3.3 X₃ has to add 0.890. EBIT/TA has the largest weight, 3.3, so profitability moves the Z-score most.

                                                                                          Worked Example: Distance to Default

                                                                                          Slide 61: Worked Example: Distance to Default
                                                                                          Slide 61

                                                                                          Worked example: distance to default

                                                                                          \[\mathrm{DD} = \frac{\ln(V_0/D) + (\mu - \sigma_V^2/2)T}{\sigma_V\sqrt{T}}, \qquad \mathrm{PD} = \Phi(-\mathrm{DD})\]
                                                                                              Question for the room ▶ raises σ_V. What is the one-year PD at σ_V = 40% (6.66% at 25%)?
                                                                                              Answer

                                                                                              20.7%: DD falls to (0.3567 + 0.05 − 0.08)/0.40 = 0.817. Volatility enters twice: it lowers the drift term and divides the distance.

                                                                                              Merton: Asset Paths and PD

                                                                                              Slide 62: Merton: Asset Paths and PD
                                                                                              Slide 62
                                                                                                Question for the room The same 40 paths, σ_V = 40% instead of 25%: how many end below the debt of 70?
                                                                                                Answer

                                                                                                8 of 40, against 3 at 25% and none at 15%; the model PD at D/V = 70% is 20.7%, 6.66% and 0.42%. With 40 paths the counts are noisy: 8/40 = 20%.

                                                                                                South German Credit (Grömping, 2019; UCI, doi:10.24432/C5QG88): 1,000 consumer credits of 1973–1975, 300 bad; stratified 700/300 split (seed 2026). Taiwan credit card data (Yeh and Lien, 2009; UCI, doi:10.24432/C55S3H): 30,000 clients, test sample of 9,000. Both data sets are saved in data/credit. From the South German Credit records the browser computes the WoE tables, IV, the logit and LDA models, the scorecard, the confusion matrices, ROC, AUC, DeLong, Gini, CAP, KS, Brier and Hosmer–Lemeshow; from the PD, outcome, sex and age of the 9,000 Taiwan test clients, the calibration and the fairness figures. The AUC of the 100 cross-validation folds, the Merton paths (seed 2026) and the reject-inference experiment come from the code of the lecture. Other variables, samples, cut-offs, costs, confidence levels, approval and acceptance shares (50, 60, 80 and 90%) are extra runs of the same code.