Evaluation, and How It Lies to You — Python Primer

Anyone can fit a model. Knowing whether the number it produced means anything is the actual skill, and it is what separates someone who can be trusted with a decision from someone who cannot.

If it looks too good to be true, it is. In this field that idiom is not world-weary caution, it is a load-bearing diagnostic, and this chapter is largely about why.

This chapter is about the specific mechanisms by which an evaluation score becomes dishonest without anyone intending it.

Three Splits, Three Jobs

flowchart LR
    D["All your data"] --> TR["<b>Train</b> ~60%<br/>the model learns<br/>from this"]
    D --> VA["<b>Validation</b> ~20%<br/>you make decisions<br/>using this"]
    D --> TE["<b>Test</b> ~20%<br/>looked at ONCE,<br/>at the very end"]
    style TE fill:#f7e8e8,stroke:#c02020
    style TR fill:#eef1fc,stroke:#2141c8

Train is what the model fits on.

Validation is where you compare models, tune hyperparameters, and decide anything. In practice cross-validation replaces a fixed validation set.

Test exists to give one honest number at the end.

Here is the part people find hard to accept. Every time you look at test performance and then change something, you have used the test set to make a decision, and it stops being an unbiased estimate. Do that ten times and your test set has quietly become a validation set, with no warning and no way to recover it.

If you need to compare many things, cross-validate on the training data. Touch the test set when you are finished, once, and report what it says even if you do not like it.

Cross-Validation

Rather than one validation split, rotate through several.

from sklearn.model_selection import cross_validate, StratifiedKFold

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_validate(pipeline, X_train, y_train, cv=cv,
                        scoring='roc_auc', return_train_score=True)

print(f"{scores['test_score'].mean():.3f} ± {scores['test_score'].std():.3f}")

Report the spread. A model scoring 0.84 ± 0.01 and one scoring 0.85 ± 0.06 are not meaningfully different, and the second is less trustworthy. A large spread usually means too little data, or folds that differ in composition.

return_train_score=True costs nothing and diagnoses a great deal:

Train	Validation	Diagnosis
High	Much lower	Overfitting. Simplify, regularise, or get more data
Low	Low	Underfitting, or the features do not contain the signal
High	Close to train	Healthy, or leakage. Check
Suspiciously perfect	Suspiciously perfect	Leakage. Almost certainly

That last row deserves emphasis. A model that performs perfectly has a bug. Real problems have irreducible noise. When an AUC comes back at 0.99 on a business problem, the correct reaction is suspicion, not celebration.

Leakage

Leakage is when information that would not be available at prediction time reaches the model during training. It is the single most common cause of a model that validates well and fails in production.

Five kinds, roughly in order of how often they occur.

1. Preprocessing before splitting

# WRONG — the scaler saw the validation rows
X_scaled = StandardScaler().fit_transform(X)
scores = cross_val_score(model, X_scaled, y, cv=5)

# RIGHT — the scaler refits inside every fold
pipe = Pipeline([('scale', StandardScaler()), ('model', model)])
scores = cross_val_score(pipe, X, y, cv=5)

The same applies to imputation, encoding, and any feature selection. Anything with a fit method belongs inside the Pipeline. This is why the scikit-learn chapter spends so long on pipelines: they make this mistake structurally impossible.

2. Target leakage from features

A feature that is a consequence of the outcome rather than a cause.

# Predicting churn, with a feature called days_since_cancellation
# Only customers who churned have a cancellation date.

The test that catches this: would this value exist, with this value, at the moment I need the prediction? Ask it of every feature. It costs ten minutes and it catches most of these.

Watch for suspiciously predictive single features:

for col in X.columns:
    auc = roc_auc_score(y, X[col].rank())
    if auc > 0.9 or auc < 0.1:
        print(f"{col}: AUC {auc:.3f}  ← investigate this")

A single raw feature that nearly solves the problem is almost always leakage.

3. Duplicate rows across folds

If the same customer appears in both training and validation, the model has effectively memorised the answer.

from sklearn.model_selection import GroupKFold

cv = GroupKFold(n_splits=5)
scores = cross_val_score(pipe, X, y, cv=cv, groups=df.customer_id)

GroupKFold guarantees all rows for one group land in the same fold. Use it whenever rows are not independent: repeated customers, multiple images of one patient, several sessions from one user.

4. Temporal leakage

Random shuffling of time-ordered data trains on the future.

from sklearn.model_selection import TimeSeriesSplit

cv = TimeSeriesSplit(n_splits=5)     # each fold trains only on earlier data

flowchart TB
    subgraph W["Random split on temporal data"]
        W1["train: Jan, Mar, May"] --> W2["predict: Feb, Apr<br/><i>using knowledge<br/>of what came after</i>"]
    end
    subgraph R["TimeSeriesSplit"]
        R1["train: Jan–Mar"] --> R2["predict: Apr"]
        R2 --> R3["train: Jan–Apr"] --> R4["predict: May"]
    end
    style W fill:#f7e8e8,stroke:#c02020
    style R fill:#eef1fc,stroke:#2141c8

If your model will predict the future in production, evaluate it predicting the future in development.

5. Target encoding computed on all the data

Covered in the categories chapter. Computing group means using each row’s own target puts a smeared copy of the answer into the feature.

Choosing a Metric

The metric encodes what you care about. Choosing it thoughtlessly is choosing a goal thoughtlessly.

Accuracy is usually the wrong choice

# 1% of transactions are fraud
y_pred = np.zeros(len(y))          # predict "never fraud" for everything
accuracy_score(y, y_pred)          # 0.99

Ninety-nine per cent accurate and completely useless. On imbalanced problems, which is most interesting problems, accuracy rewards ignoring the minority class.

The confusion matrix

Everything else is built from four counts.

from sklearn.metrics import confusion_matrix, classification_report

confusion_matrix(y_true, y_pred)
print(classification_report(y_true, y_pred, digits=3))

	Predicted negative	Predicted positive
Actually negative	True negative	False positive
Actually positive	False negative	True positive

Precision is: of those I flagged, how many were right? Use it when a false positive is expensive, such as blocking a legitimate customer.

Recall is: of those that were real, how many did I catch? Use it when a false negative is expensive, such as missing a disease.

You trade one against the other. Which matters is a business question, and someone has to answer it before you can choose a threshold.

The threshold is a separate decision

Most classifiers produce a probability. Turning it into a decision requires a cutoff, and 0.5 is a default, not an answer.

proba = model.predict_proba(X_test)[:, 1]

for t in [0.1, 0.3, 0.5, 0.7, 0.9]:
    pred = (proba >= t).astype(int)
    print(f"threshold {t}: precision {precision_score(y_test, pred):.3f}, "
          f"recall {recall_score(y_test, pred):.3f}")

Pick the threshold using the costs. If a missed fraud costs £500 and an unnecessary review costs £5, the maths tells you where to put it.

ROC-AUC and average precision

roc_auc_score(y_test, proba)              # ranking quality across all thresholds
average_precision_score(y_test, proba)    # area under precision-recall

On imbalanced data, prefer average precision. ROC-AUC can look respectable while the model is useless on the rare class, because the enormous number of true negatives dominates the calculation.

Regression metrics

mean_absolute_error(y_test, pred)                    # in original units
np.sqrt(mean_squared_error(y_test, pred))            # penalises large errors
mean_absolute_percentage_error(y_test, pred)         # relative
r2_score(y_test, pred)                               # proportion of variance

Mean absolute error is the most interpretable: “on average we are off by £43”. Root mean squared error punishes large misses more, which is right when one big error is worse than several small ones.

Avoid percentage error when the true values can be near zero. It explodes.

Calibration

A probability that feeds into a decision must be right, not merely well-ordered.

If your model says 30%, do about 30% of those cases actually happen? A model can rank perfectly and still be systematically overconfident, which matters enormously when the probability gets multiplied by money.

from sklearn.calibration import calibration_curve

prob_true, prob_pred = calibration_curve(y_test, proba, n_bins=10)

fig, ax = plt.subplots()
ax.plot([0, 1], [0, 1], '--', color='grey', label='perfect')
ax.plot(prob_pred, prob_true, 'o-', label='model')
ax.set_xlabel('predicted probability')
ax.set_ylabel('observed frequency')
ax.legend()

Points below the diagonal mean overconfidence. Fix it by wrapping the model:

from sklearn.calibration import CalibratedClassifierCV
calibrated = CalibratedClassifierCV(model, method='isotonic', cv=5)

Ranking metrics like AUC are completely blind to calibration. If your output is a probability anyone will act on, check it.

Always Beat a Baseline

from sklearn.dummy import DummyClassifier, DummyRegressor

DummyClassifier(strategy='prior')       # always predict the majority class
DummyRegressor(strategy='mean')         # always predict the average

Three lines. If your carefully tuned model cannot beat this by a margin you would defend in a meeting, you have learned something valuable and cheap.

Add a simple business baseline too. “The rule the team currently uses” is the comparison that actually matters, and a model that fails to beat it should not ship regardless of its AUC.

Decide Before You Look

The habit that prevents most self-deception. Before running anything, write down:

The metric, and why it matches the business cost
The splitting strategy, including groups and time
The baseline you must beat
What result would make you abandon the approach

That last one is the hardest and the most valuable. A project with no failure condition never fails; it just continues.

Keep the note. When you find yourself trying a fourth metric because the first three were disappointing, it will tell you what you decided when you were being honest.

What Carries Forward

Look at the test set once. Every peek that changes a decision spends its value.

Anything that learns goes inside the Pipeline. This alone prevents the most common leakage.

Perfect scores mean a bug. Investigate rather than celebrate.

Accuracy is the wrong metric for imbalanced problems. Use average precision.

Check calibration if anyone will act on the probability.

Beat a dummy baseline first, and the current business rule second.

Next: feature engineering, where most real model improvement actually comes from.