scikit-learn: The Interface, Not the Algorithms

This chapter has a deliberate and slightly unusual boundary, so let me state it before we start.

It teaches you the scikit-learn interface. It does not teach you what the algorithms do.

That is not an oversight. The interface is a Python skill: objects, conventions, composition, and a particular design that makes a whole class of mistake impossible. You can learn it in an afternoon and it applies to every model in the library, including ones that have not been written yet. How a gradient boosting machine actually works is a different subject, it takes considerably longer, and it belongs in Learn ML Algorithms rather than in a primer.

The division is worth stating plainly because it maps onto how the work actually splits:

Question	Where it belongs
How do I fit a model to this data?	Here
How do I avoid leaking the test set into training?	Here, and it is the point of this chapter
How do I search hyperparameters without cheating?	Here
How does a random forest reduce variance?	The algorithms book
Why does the kernel trick work?	The algorithms book
Should I use boosting or a linear model?	The algorithms book

By the end of this chapter you will be able to build, validate and persist a model correctly without knowing what is inside it. That sounds like it should be dangerous. In practice it is the right order to learn things, because the machinery that keeps you honest is what stops a beginner producing confidently wrong results.

One Interface, Every Model

The design insight that made scikit-learn win is that everything is the same shape. Every model, every preprocessing step, every feature selector implements the same small contract.

flowchart TB
    subgraph E["Every estimator"]
        F["<b>fit(X, y)</b><br/>learn from data<br/>returns self"]
    end
    F --> P["<b>predict(X)</b><br/>models only"]
    F --> T["<b>transform(X)</b><br/>preprocessors only"]
    F --> S["<b>score(X, y)</b><br/>models only"]
    T --> FT["<b>fit_transform(X, y)</b><br/>fit then transform,<br/>often faster together"]
    style E fill:#eef1fc,stroke:#2141c8,stroke-width:2px

Which means this code is identical apart from one line:

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC

for model in [LogisticRegression(), RandomForestClassifier(), SVC()]:
    model.fit(X_train, y_train)
    print(type(model).__name__, model.score(X_test, y_test))

Three completely unrelated algorithms, one loop. You can swap a model without rewriting anything around it, which makes comparison cheap and makes the honest baseline habit easy to sustain.

Two conventions worth internalising

Hyperparameters go in the constructor. Learned state comes out with a trailing underscore.

model = RandomForestClassifier(n_estimators=200, max_depth=5)  # your choices
model.fit(X_train, y_train)

model.n_estimators      # 200 — what you asked for
model.feature_importances_   # trailing underscore — learned from data
model.classes_               # learned

That underscore is a genuine convention, not decoration. If an attribute ends in one, it did not exist before fit was called. If you access it early you get an AttributeError, which is the library telling you the object is not ready.

fit always returns self, which is what allows chaining and what makes the pipeline machinery work.

Splitting, and Doing It Once

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.2,
    random_state=42,      # reproducibility
    stratify=y,           # keep class proportions in both halves
)

stratify=y matters more than people expect. On an imbalanced problem, a random split can hand you a test set with a noticeably different positive rate than the training set, and your metric then measures the split rather than the model.

The test set has one job: to be looked at once, at the very end. Every time you check test performance and then change something, you have used the test set to make a decision, and it is no longer an unbiased estimate of anything. Use cross-validation on the training data for all your decisions, and touch the test set when you are finished.

The Pipeline, and Why It Is the Point of This Chapter

Here is the most expensive mistake in applied machine learning, written out:

# WRONG. This looks completely reasonable and it is not.
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)          # ← fitted on ALL the data

X_train, X_test, y_train, y_test = train_test_split(X_scaled, y)
model.fit(X_train, y_train)
model.score(X_test, y_test)                 # optimistic, and you cannot tell

The scaler computed a mean and standard deviation using every row, including the rows that later became the test set. Information about the test set has flowed into the training process. The score comes out slightly too high, nothing warns you, and the model underperforms in production by an amount nobody can explain.

This is called leakage, and scaling before splitting is only its most common form. Imputing missing values, selecting features by correlation with the target, and target encoding all leak the same way.

Pipeline fixes it structurally, which is far better than fixing it by remembering.

flowchart TB
    subgraph BAD["Manual — leakage is one line away"]
        B1["scale all data"] --> B2["split"] --> B3["fit model"]
        B3 --> B4["score is optimistic<br/>and silent about it"]
    end
    subgraph GOOD["Pipeline — leakage is impossible"]
        G1["split first"] --> G2["Pipeline.fit(train)"]
        G2 --> G3["scaler learns from<br/>the training fold only"]
        G3 --> G4["model fits on<br/>the scaled training fold"]
        G4 --> G5["Pipeline.predict(test)<br/>applies the SAME<br/>learned scaling"]
    end
    style BAD fill:#f7e8e8,stroke:#c02020
    style GOOD fill:#eef1fc,stroke:#2141c8

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipe = Pipeline([
    ('scale', StandardScaler()),
    ('model', LogisticRegression(max_iter=1000)),
])

pipe.fit(X_train, y_train)       # scaler fits on train only, then the model
pipe.predict(X_test)             # scaler transforms using the training stats

A Pipeline is itself an estimator. It has fit, predict and score, so anything that accepts a model accepts a pipeline. That is the property that makes the next section work.

The rule to carry away: nothing that learns from data should ever run outside a Pipeline. If a step has a fit method, it belongs inside.

ColumnTransformer: Different Columns, Different Treatment

Real data is not one type. You have numbers that need scaling, categories that need encoding, and text that needs something else entirely.

flowchart LR
    X["Input DataFrame"] --> CT{ColumnTransformer}
    CT -->|"age, income"| N["impute median<br/>→ StandardScaler"]
    CT -->|"city, tier"| C["impute constant<br/>→ OneHotEncoder"]
    CT -->|"notes"| D["drop"]
    N --> M["hstack → one matrix"]
    C --> M
    M --> EST["Estimator"]
    style CT fill:#eef1fc,stroke:#2141c8

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
from sklearn.impute import SimpleImputer

numeric = ['age', 'income']
categorical = ['city', 'tier']

preprocess = ColumnTransformer([
    ('num', Pipeline([
        ('impute', SimpleImputer(strategy='median')),
        ('scale', StandardScaler()),
    ]), numeric),
    ('cat', Pipeline([
        ('impute', SimpleImputer(strategy='constant', fill_value='missing')),
        ('encode', OneHotEncoder(handle_unknown='ignore', sparse_output=False)),
    ]), categorical),
], remainder='drop')

full = Pipeline([
    ('prep', preprocess),
    ('model', LogisticRegression(max_iter=1000)),
])

Two arguments in there are doing quiet, important work.

handle_unknown='ignore' decides what happens when a category appears in production that was not in training. Without it, the pipeline raises and your service returns a 500. With it, the unseen category encodes as all zeros and you get a prediction. Neither is obviously correct, but you should choose deliberately rather than discover it at three in the morning.

remainder='drop' is the default and it is the safe one. It means any column you did not name is discarded. The alternative, 'passthrough', silently feeds unlisted columns straight into the model, which is how a customer ID ends up as a feature.

sparse_output=False is there for the next paragraph, and it deserves a warning of its own.

For readable output, ask for DataFrames back:

preprocess.set_output(transform='pandas')

You keep column names all the way through, which turns debugging from guesswork into reading. This is also the most common first error people hit with ColumnTransformer. OneHotEncoder returns a sparse matrix by default, because one-hot output is mostly zeros and sparse storage is dramatically cheaper on high-cardinality columns. A pandas DataFrame cannot hold sparse data, so combining the two raises:

ValueError: Pandas output does not support sparse data.

Set sparse_output=False and the problem goes away, at the cost of materialising every zero. On a few dozen categories that is free. On fifty thousand product codes it is not, and there you should drop the pandas output and work with the sparse matrix instead. Column names are a debugging convenience; memory is a constraint.

Cross-Validation Without Cheating

Because a pipeline is an estimator, you can hand the whole thing to cross-validation, and every preprocessing step is then refitted inside each fold.

from sklearn.model_selection import cross_validate, StratifiedKFold

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

scores = cross_validate(
    full, X_train, y_train,
    cv=cv,
    scoring=['accuracy', 'roc_auc', 'average_precision'],
    return_train_score=True,
)

print(scores['test_roc_auc'].mean(), '±', scores['test_roc_auc'].std())

Two things worth noticing.

Report the spread, not just the mean. A model scoring 0.84 with a standard deviation of 0.01 and one scoring 0.85 with a standard deviation of 0.06 are not meaningfully different, and the second is less trustworthy.

return_train_score=True is a cheap diagnostic. Training score far above validation score means overfitting. Both low means underfitting. Both similar and low means the features do not contain the signal, and no amount of model selection will help.

For anything with a time dimension, a random split is training on the future:

from sklearn.model_selection import TimeSeriesSplit
cv = TimeSeriesSplit(n_splits=5)     # each fold trains on the past only

Hyperparameter Search

Double underscores address a step inside a pipeline, and the syntax nests as deep as your pipeline does.

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform

search = RandomizedSearchCV(
    full,
    {
        'model__C': loguniform(1e-3, 1e2),
        'prep__num__impute__strategy': ['median', 'mean'],
    },
    n_iter=40,
    cv=cv,
    scoring='roc_auc',
    n_jobs=-1,
    random_state=42,
)
search.fit(X_train, y_train)

search.best_params_
search.best_score_
best = search.best_estimator_        # already refitted on all training data

Prefer RandomizedSearchCV over GridSearchCV as a default. Grid search spends its budget evenly across a space where most parameters do not matter, while random search covers the important dimensions far better for the same number of fits.

The whole search happens inside the pipeline, so preprocessing is refitted for every fold of every candidate. That is the correct and expensive thing to do, and it is why doing it wrong is so tempting.

Custom Steps

When you need a transformation the library does not have, you have two options and the first covers most cases.

from sklearn.preprocessing import FunctionTransformer
import numpy as np

log_transform = FunctionTransformer(np.log1p, feature_names_out='one-to-one')

For anything that must learn something during fit, write a class:

from sklearn.base import BaseEstimator, TransformerMixin

class ClipOutliers(BaseEstimator, TransformerMixin):
    """Clip each column to percentiles learned from the training data."""

    def __init__(self, lower=0.01, upper=0.99):
        self.lower = lower          # store arguments unchanged, no validation
        self.upper = upper          # here — sklearn clones estimators by
                                    # reading these attributes back

    def fit(self, X, y=None):
        self.lo_ = np.quantile(X, self.lower, axis=0)
        self.hi_ = np.quantile(X, self.upper, axis=0)
        return self                 # always return self

    def transform(self, X):
        return np.clip(X, self.lo_, self.hi_)

Follow the conventions exactly and your class works everywhere in the library: inside pipelines, inside cross-validation, inside hyperparameter search. Deviate and you get confusing failures during cloning.

Persisting a Model

import joblib

joblib.dump(best, 'model-v3.joblib')
loaded = joblib.load('model-v3.joblib')

Three warnings, and the first one is serious.

A pickle is executable code. Loading one runs whatever is inside it. Never load a model file from a source you do not trust.

Versions must match. A model saved under one scikit-learn version may fail or, worse, behave differently under another. Pin your versions and record the version alongside the model file.

Save the whole pipeline, not just the model. If you persist the estimator without its preprocessing, whoever deploys it has to reconstruct the transformations by hand, and they will get one of them subtly wrong.

joblib.dump({
    'pipeline': best,
    'sklearn_version': sklearn.__version__,
    'trained_at': datetime.now(timezone.utc).isoformat(),
    'feature_names': list(X_train.columns),
    'cv_score': search.best_score_,
}, 'model-v3.joblib')

The Whole Thing, End to End

# 1. Split once, stratified.
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)

# 2. Everything that learns goes in the pipeline.
pipe = Pipeline([('prep', preprocess), ('model', LogisticRegression(max_iter=1000))])

# 3. Compare against a trivial baseline. Frequently it wins.
from sklearn.dummy import DummyClassifier
base = cross_validate(DummyClassifier(strategy='prior'), X_train, y_train,
                      cv=cv, scoring='roc_auc')['test_score'].mean()

# 4. Search on the training data only.
search.fit(X_train, y_train)

# 5. Touch the test set once, at the end.
final = search.best_estimator_.score(X_test, y_test)

print(f'baseline {base:.3f} | cv {search.best_score_:.3f} | test {final:.3f}')

Step three is the one people skip. DummyClassifier predicts the majority class and nothing else. If your carefully tuned model cannot beat it by a margin you would defend in a meeting, you have learned something valuable and cheap.

What This Chapter Deliberately Left Out

Everything about how the models work. Which algorithm suits which problem, why regularisation controls overfitting, what a decision boundary looks like, how boosting differs from bagging.

That material is a book rather than a chapter, and it is the next one. What you have now is the scaffolding that makes learning it safe: you can fit any model correctly, validate it honestly, and ship it without leaking. The algorithms slot into that scaffolding one at a time.

What Carries Forward

One contract covers everything. Fit, transform, predict. Learn it once.

Pipelines are not tidiness, they are correctness. They make the most expensive mistake in this field structurally impossible, which is much better than making it merely avoidable.

The test set is looked at once. Every peek that changes a decision spends a little of its value.

Always beat a dumb baseline before believing anything. It costs three lines and it has saved more projects than any modelling technique.