Categories and Encoding — Python Primer

A category is a value from a fixed set: a city, a tier, a status, a product line. Models cannot consume text, so at some point every category becomes a number.

How you do that conversion matters more than beginners expect, and one popular method quietly cheats.

The scale of the problem is easy to underestimate. A retailer’s product catalogue is not three categories, it is several hundred thousand, and the naive approach turns that into several hundred thousand columns. Choosing well here is the difference between a model that trains overnight and one that never trains at all.

The Categorical Dtype

Before encoding, store categories properly. Pandas has a dedicated type that keeps one copy of each distinct value and stores integer codes for the rest.

df['tier'] = df.tier.astype('category')

The memory difference is not marginal:

s = pd.Series(['gold', 'silver', 'bronze'] * 100_000)

s.memory_usage(deep=True)                    # ~18,000,000 bytes
s.astype('category').memory_usage(deep=True) #    ~100,000 bytes

Roughly a 180-fold reduction on this column. The rule of thumb: convert any text column where the number of distinct values is small relative to the number of rows.

for c in df.select_dtypes(['object', 'string']):
    ratio = df[c].nunique() / len(df)
    if ratio < 0.5:
        df[c] = df[c].astype('category')
        print(f"{c}: {df[c].nunique()} categories")

Ordered categories

Some categories have a natural order, and telling pandas unlocks comparison and sensible sorting:

from pandas.api.types import CategoricalDtype

sizes = CategoricalDtype(['small', 'medium', 'large'], ordered=True)
df['size'] = df['size'].astype(sizes)

df[df['size'] > 'small']            # works, because the order is declared
df.sort_values('size')              # small, medium, large — not alphabetical

Without this, sorting gives you large, medium, small, which is alphabetical and useless.

Two traps

Filtering keeps unused categories. The category list is part of the type, not the data.

sub = df[df.tier == 'gold']
sub.tier.cat.categories        # still lists silver and bronze
sub.groupby('tier').size()     # rows of zero for the absent ones

Fix with observed=True on the groupby, or remove_unused_categories().

Concatenating frames with different category sets produces an object column, silently losing the type. Set the categories explicitly on both sides first.

Encoding for Models

Four approaches. Choosing between them is mostly about how many distinct values you have.

flowchart TD
    Q{"How many<br/>distinct values?"}
    Q -->|"2"| B["<b>Binary</b><br/>0 / 1"]
    Q -->|"a handful,<br/>no order"| O["<b>One-hot</b><br/>a column each"]
    Q -->|"a handful,<br/>with order"| R["<b>Ordinal</b><br/>map to 0,1,2..."]
    Q -->|"many<br/>(hundreds+)"| T["<b>Target encoding</b><br/><i>leaks — see below</i>"]
    style O fill:#eef1fc,stroke:#2141c8
    style T fill:#f7e8e8,stroke:#c02020

One-hot encoding

One new column per category, holding 0 or 1. The default for unordered categories.

pd.get_dummies(df, columns=['city', 'tier'], drop_first=False)

For anything going into a model, use the scikit-learn version instead, so the categories are learned from training data and applied consistently:

from sklearn.preprocessing import OneHotEncoder

enc = OneHotEncoder(handle_unknown='ignore', sparse_output=False)

handle_unknown='ignore' decides what happens when production shows you a category training never saw. Without it your service raises; with it, the unseen value encodes as all zeros. Neither is obviously right, but choose deliberately rather than at three in the morning.

drop_first=True removes one column to avoid perfect collinearity. It matters for linear models and ordinary regression. It does not matter for trees, and it makes coefficients harder to interpret, so do not apply it reflexively.

The cost of one-hot is width. Fifty thousand product codes become fifty thousand columns, which is why the next options exist.

Ordinal encoding

Map categories to integers, in a meaningful order:

order = {'small': 0, 'medium': 1, 'large': 2}
df['size_code'] = df['size'].map(order)

Only do this when the order is real. Encoding cities as 0, 1, 2, 3 tells a linear model that Sydney is halfway between Delhi and Tokyo, which is nonsense.

Tree-based models are more forgiving, because they split on thresholds rather than assuming linearity. Arbitrary integer codes often work fine with gradient boosting and badly with regression.

Frequency encoding

Replace each category with how often it appears:

freq = df.city.value_counts(normalize=True)
df['city_freq'] = df.city.map(freq)

One column regardless of cardinality, and genuinely informative when rarity matters, which it often does in fraud and anomaly work.

Target encoding, and why it leaks

Replace each category with the average target value for that category.

# WRONG — and it is not obvious why
means = df.groupby('city').churned.mean()
df['city_encoded'] = df.city.map(means)

This looks reasonable and is a serious mistake. Each row’s encoding was computed using that row’s own target value. The feature contains a smeared copy of the answer.

The consequence: cross-validation scores look excellent, and the model fails in production. Worse, the effect is strongest for rare categories. A city appearing once gets encoded as exactly that row’s outcome, which is the answer, perfectly.

flowchart TB
    subgraph BAD["Naive target encoding"]
        B1["row's own target"] --> B2["group mean"] --> B3["that row's feature"]
        B3 --> B4["the feature contains<br/>the answer"]
    end
    subgraph GOOD["Out-of-fold encoding"]
        G1["split into folds"] --> G2["for each fold:<br/>compute means from<br/>the OTHER folds"]
        G2 --> G3["encode this fold<br/>with those means"]
        G3 --> G4["no row sees<br/>its own target"]
    end
    style BAD fill:#f7e8e8,stroke:#c02020
    style GOOD fill:#eef1fc,stroke:#2141c8

The fix is out-of-fold encoding, computing each fold’s values from the other folds only:

from sklearn.model_selection import KFold
import numpy as np

def target_encode_oof(df, col, target, n_splits=5, smoothing=10):
    """Out-of-fold target encoding with smoothing toward the global mean."""
    global_mean = df[target].mean()
    encoded = pd.Series(np.nan, index=df.index)

    for train_idx, val_idx in KFold(n_splits, shuffle=True, random_state=0).split(df):
        fold = df.iloc[train_idx]
        stats = fold.groupby(col)[target].agg(['mean', 'count'])
        # shrink small groups toward the global mean
        smoothed = ((stats['mean'] * stats['count'] + global_mean * smoothing)
                    / (stats['count'] + smoothing))
        encoded.iloc[val_idx] = df.iloc[val_idx][col].map(smoothed).values

    return encoded.fillna(global_mean)

The smoothing term is the second half of the fix. A category seen twice should not be trusted as much as one seen ten thousand times, so its estimate is pulled toward the overall average. The smoothing parameter controls how hard.

Better still, use a library implementation inside a pipeline so the whole thing is refitted per fold automatically. The point of understanding the mechanism is knowing why that matters.

High Cardinality

When a column has thousands of distinct values, one-hot is impractical. The options, roughly in order of how often they are right:

Group the rare tail. Keep the top N, call everything else other. Simple and effective.

top = df.city.value_counts().nlargest(20).index
df['city_grouped'] = df.city.where(df.city.isin(top), 'other')

Frequency encoding. One column, no leakage risk.
Out-of-fold target encoding. Powerful, and only with the discipline above.
Group by a meaningful attribute instead. Replace postcode with region, product with category. Domain knowledge beats any automated scheme.
Embeddings, if you are already training a neural network. Overkill otherwise.

Ordering Matters for Charts

Categorical order affects plots as well as models:

counts = df.city.value_counts()
df['city'] = pd.Categorical(df.city, categories=counts.index, ordered=True)

Now every chart shows categories from most to least common, which is usually what a reader wants, rather than alphabetically, which is almost never what anyone wants.

What Carries Forward

Convert low-cardinality text to category. Large memory savings and clearer intent.

Declare the order when there is one. It makes sorting and comparison work.

One-hot for a handful of unordered categories, with handle_unknown='ignore'.

Never target-encode naively. Compute out of fold, and smooth small groups toward the global mean.

Group the rare tail before reaching for anything clever. It solves most high-cardinality problems.

That completes the data-wrangling section. Next: making charts that answer a question.