Some examples

The goal is to learn a decision rule \hat{f}: \mathcal{X} \to \{1, \ldots, K\} that maps features to class labels as accurately as possible.

For a binary response Y \in \{0, 1\}, logistic regression models the probability of the positive class via the sigmoid function:

P(Y = 1 \mid \mathbf{x}) = \sigma(\mathbf{x}^\top \boldsymbol{\beta}) = \frac{1}{1 + e^{-\mathbf{x}^\top \boldsymbol{\beta}}}

The decision rule is simply:

\hat{y} = \begin{cases} 1 & \text{if } P(Y=1 \mid \mathbf{x}) \geq 0.5 \\ 0 & \text{otherwise} \end{cases}

This corresponds to a linear decision boundary \mathbf{x}^\top\boldsymbol{\beta} = 0 in feature space.

Key properties

Binary logistic regression generalises to K > 2 classes in two main ways:

• One-vs-Rest (OvR): fit K separate binary classifiers, each asking “is this observation class k or not?”. Predict the class whose classifier returns the highest probability.

• Multinomial (softmax): fit all classes simultaneously using a single model:

P(Y = k \mid \mathbf{x}) = \frac{\exp(\mathbf{x}^\top \boldsymbol{\beta}_k)}{\displaystyle\sum_{j=1}^{K} \exp(\mathbf{x}^\top \boldsymbol{\beta}_j)}

sklearn’s LogisticRegression uses the multinomial (softmax) approach by default when K > 2. The decision boundary remains linear — each class is still separated by a hyperplane.

Source: Kaggle - Credit Score Classification dataset

The business problem

• A bank wants to automatically assign each customer to a credit score band:

  • Good — low credit risk; eligible for favourable rates
  • Standard — moderate risk; standard products
  • Poor — high risk; restricted lending

The data

• 100,000 labelled records across 8 months
• 27 raw features: income, payment history, loan counts, credit utilisation, …
• Each customer appears in multiple monthly snapshots

raw = pd.read_csv('data/train.csv', low_memory=False)

• We print the shape of the data:

print('Train Data Size : ',raw.shape)

Train Data Size :  (100000, 28)

• We also check the first few rows to get a sense of the data:

raw.info()

<class 'pandas.DataFrame'>
RangeIndex: 100000 entries, 0 to 99999
Data columns (total 28 columns):
 #   Column                    Non-Null Count   Dtype  
---  ------                    --------------   -----  
 0   ID                        100000 non-null  str    
 1   Customer_ID               100000 non-null  str    
 2   Month                     100000 non-null  str    
 3   Name                      90015 non-null   str    
 4   Age                       100000 non-null  str    
 5   SSN                       100000 non-null  str    
 6   Occupation                100000 non-null  str    
 7   Annual_Income             100000 non-null  str    
 8   Monthly_Inhand_Salary     84998 non-null   float64
 9   Num_Bank_Accounts         100000 non-null  int64  
 10  Num_Credit_Card           100000 non-null  int64  
 11  Interest_Rate             100000 non-null  int64  
 12  Num_of_Loan               100000 non-null  str    
 13  Type_of_Loan              88592 non-null   str    
 14  Delay_from_due_date       100000 non-null  int64  
 15  Num_of_Delayed_Payment    92998 non-null   str    
 16  Changed_Credit_Limit      100000 non-null  str    
 17  Num_Credit_Inquiries      98035 non-null   float64
 18  Credit_Mix                100000 non-null  str    
 19  Outstanding_Debt          100000 non-null  str    
 20  Credit_Utilization_Ratio  100000 non-null  float64
 21  Credit_History_Age        90970 non-null   str    
 22  Payment_of_Min_Amount     100000 non-null  str    
 23  Total_EMI_per_month       100000 non-null  float64
 24  Amount_invested_monthly   95521 non-null   str    
 25  Payment_Behaviour         100000 non-null  str    
 26  Monthly_Balance           98800 non-null   str    
 27  Credit_Score              100000 non-null  str    
dtypes: float64(4), int64(4), str(20)
memory usage: 21.4 MB

• Several numeric columns are stored as strings with dirty values (_, trailing letters, embedded underscores).
• Data is missing in many columns, but not with a consistent placeholder.
• Cleaning is needed before any modelling.

The raw data required several steps before modelling:

• Dropped identifier and sensitive columns: ID, Customer_ID, Name, SSN, Type_of_Loan
• Coerced dirty numeric strings: stripped embedded underscores and trailing characters from 8 columns
• Parsed credit history age: "22 Years and 1 Months" → integer months (Credit_History_Months)
• Replaced _ placeholder with NaN in Credit_Mix

Accuracy — the simplest metric: fraction of all predictions that are correct.

\text{Accuracy} = \frac{\text{Number correct}}{\text{Total predictions}}

But accuracy can be misleading. In our dataset, 53 % of customers are “Standard” — a classifier that always predicts “Standard” scores 53 % accuracy without learning anything useful.

For each class k we also compute:

• Precision - Of all customers we predicted as k, what fraction truly are k?
• Recall - Of all customers that truly are k, what fraction did we identify?
• F1 - Harmonic mean of precision and recall — balances the two.

When classes are imbalanced, a model can game accuracy by favouring the majority class. Precision and recall expose this — they measure performance within each class separately.

For our credit score problem (K = 3):

For each class k, treating it as “positive” vs. “all others”:

\text{Precision}_k = \frac{TP_k}{TP_k + FP_k} \qquad \text{Recall}_k = \frac{TP_k}{TP_k + FN_k}

F_{1,k} = \frac{2 \cdot \text{Precision}_k \cdot \text{Recall}_k}{\text{Precision}_k + \text{Recall}_k}

• Precision: of all customers predicted as class k, what fraction truly belong to k? High precision means few false alarms.
• Recall: of all customers who truly belong to class k, what fraction did we correctly identify? High recall means few missed cases.
• F1: the harmonic mean of precision and recall — useful when you care about both equally, and penalises extreme imbalances between the two.

In credit scoring, recall for “Poor” is critical — a missed high-risk customer can be very costly. But precision also matters — falsely flagging a “Good” customer harms trust.

Weighted average is usually most informative when class sizes differ — as they do here (Good is the rarest band).

A decision tree partitions the feature space into rectangular regions using a recursive sequence of binary splits. Each leaf is assigned the majority class of training observations that land there.

Decision trees are fully non-parametric — no distributional assumption about the boundary shape. The tree structure is the model.

At each node we search for the feature j and threshold t that most reduce impurity in the resulting child nodes.

The Gini gain of a candidate split is:

\Delta G = G(\text{parent}) - \frac{n_L}{n}\,G(\text{left}) - \frac{n_R}{n}\,G(\text{right})

We choose (j^*, t^*) that maximises \Delta G.

A decision tree is grown recursively:

1. Find the best split (j^*, t^*) at the current node.
2. Partition observations into left (x_{j^*} \leq t^*) and right (x_{j^*} > t^*) children.
3. Recurse on each child.
4. Stop when a stopping criterion is met.

Stopping criteria

The stopping criteria above are pre-pruning controls — they prevent the tree from growing too large in the first place.

Post-pruning takes the opposite approach: grow the full tree first, then cut back branches whose impurity reduction is outweighed by the cost of keeping them.

The optimal \alpha is chosen by evaluating held-out performance across the pruning path.

print(classification_report(
    y_val, y_pred_dt,
    target_names=['Poor', 'Standard', 'Good'],
))

              precision    recall  f1-score   support

        Poor       0.72      0.69      0.70      5799
    Standard       0.75      0.74      0.75     10635
        Good       0.58      0.65      0.61      3566

    accuracy                           0.71     20000
   macro avg       0.68      0.69      0.69     20000
weighted avg       0.71      0.71      0.71     20000

• We see that the tree performs reasonably well on “Poor” and “Standard” classes, but struggles with “Good” — likely due to overlap with “Standard” and the linear splits.
• It has higher recall for “Poor” than logistic regression, but lower precision — it flags more high-risk customers but also more false alarms.

Decision trees have high variance: small changes in the training data can produce very different trees.

Why?

• Each split depends entirely on which observations happened to be in the training set.
• A single influential observation can redirect an entire branch.
• Deep trees memorise noise rather than signal.

Can we reduce variance without substantially increasing bias?

Idea: train many trees on different bootstrap samples of the training data, then combine their predictions by majority vote.

• Each tree sees \approx 63\% of observations; the rest form a natural out-of-bag (OOB) validation set.
• Averaging over B trees reduces variance by a factor of \approx B — provided trees are uncorrelated.

The problem with plain bagging: if one feature is very strong, all trees split on it first → trees are correlated → variance reduction is limited.

• Decorrelates trees → greater variance reduction than plain bagging.
• Slight bias increase (fewer features considered per split), but usually a net improvement.
• The OOB observations give a free internal accuracy estimate — no separate validation set needed.

Random forests provide a natural variable importance measure:

Caveat: MDI can overstate importance for high-cardinality or continuous features. Permutation importance is a more reliable alternative for such cases.

📅 What’s Next?

• Lecture 17: Time Series — stationarity, autocorrelation, ARIMA, and forecasting with a real economic dataset.
• Lectures 18–20: Unsupervised learning, neural networks, and ethics in data science.