AI/Biz

Python Primer·Chapter 23

Feature Engineering That Survives

Most model improvement comes from better features, not better algorithms. The techniques that generalise, and the specific ways a feature that works in a notebook turns out to be impossible to compute in production.

Given a fixed dataset, swapping algorithms usually moves your metric by a small amount. Constructing a feature that captures something real about the problem can move it a great deal.

This is also where domain knowledge beats technique, which is good news, because you probably know more about your business than about gradient boosting.

Ask the Production Question First

Before building any feature, answer this:

Would this value exist, with this value, at the moment I need the prediction?

Most leakage is a failure to ask it. A feature computed from the whole dataset, or from events after the prediction point, or from a table that is only populated later, will validate beautifully and fail in production.

flowchart LR
    T1["customer<br/>signs up"] --> T2["<b>PREDICTION<br/>POINT</b><br/>what do we know<br/><i>right now</i>?"]
    T2 --> T3["they churn<br/>or do not"]
    T2 -.->|"anything from here<br/>is LEAKAGE"| T3
    style T2 fill:#eef1fc,stroke:#2141c8,stroke-width:2px

Three questions that catch nearly all of it:

  • When is this computed? If after the outcome, it is leakage.
  • Where does it come from? If from a table updated by the outcome, it is leakage.
  • Is it available with the same latency in production? A feature that takes six hours to compute cannot serve a real-time request.

Numeric Transformations

Log for skewed positive values. Revenue, income, session length, counts.

df['log_revenue'] = np.log1p(df.revenue)      # log1p handles zero safely

This compresses a long right tail into something more symmetric. It helps linear models considerably and tree models not at all, since trees split on order and a monotone transform does not change the order.

The most famous example in retail is a ratio and a set of timing features. Ever wonder how Target and Walmart know to email you about garden furniture the same week you started thinking about your garden? Nobody read your mind. Somebody noticed that a shift in what you buy, and how often, predicts a change in circumstances better than any single purchase does. That is feature engineering, and it is worth more than the algorithm sitting behind it.

Ratios are often more meaningful than their components.

df['revenue_per_order'] = df.revenue / df.orders.replace(0, np.nan)
df['discount_rate'] = df.discount / df.list_price

Guard against division by zero, and think about what a missing result should mean.

Binning turns a continuous variable into ranges, which can capture a non-linear relationship a linear model would miss.

df['age_band'] = pd.cut(df.age, bins=[0, 25, 35, 50, 65, 120],
                        labels=['<25', '25-34', '35-49', '50-64', '65+'])
df['revenue_decile'] = pd.qcut(df.revenue, 10, labels=False, duplicates='drop')

cut uses the boundaries you give. qcut uses quantiles, giving equal-sized groups. Prefer meaningful boundaries when the domain has them, because “under 25” is explainable and “decile 3” is not.

Binning throws away information, so it is a trade. Use it when the relationship really is stepped, or when you need explainability.

Aggregation Features

The single most productive family in business data: compare each row against its own group.

# The group's average, attached to every row in that group
df['tier_avg_revenue'] = df.groupby('tier').revenue.transform('mean')

# How this row differs from its peers
df['vs_tier'] = df.revenue - df.tier_avg_revenue
df['vs_tier_ratio'] = df.revenue / df.tier_avg_revenue

# Rank within the group
df['rank_in_tier'] = df.groupby('tier').revenue.rank(pct=True)

transform returns one value per original row rather than one per group, which is what makes this work in a single line.

The reason these help: absolute revenue of £500 means nothing on its own. Being twice the average for your segment means a great deal. You have given the model context it could not derive itself.

For customer-level aggregates from transaction data:

agg = orders.groupby('customer_id').agg(
    n_orders=('order_id', 'count'),
    total_revenue=('revenue', 'sum'),
    avg_revenue=('revenue', 'mean'),
    revenue_std=('revenue', 'std'),
    first_order=('created_at', 'min'),
    last_order=('created_at', 'max'),
    n_categories=('category', 'nunique'),
)
agg['days_active'] = (agg.last_order - agg.first_order).dt.days
agg['orders_per_month'] = agg.n_orders / (agg.days_active / 30).clip(lower=1)

Time-Based Features, Without Looking Forward

This is where leakage is easiest to introduce and hardest to spot.

df = df.sort_values(['customer_id', 'date'])

# Correct: shift(1) means "the previous row", strictly the past
df['prev_revenue'] = df.groupby('customer_id').revenue.shift(1)

# Correct: rolling window that ends before the current row
df['revenue_7d_avg'] = (df.groupby('customer_id').revenue
                          .transform(lambda s: s.shift(1).rolling(7, min_periods=1).mean()))

# WRONG: includes the current row
df['bad'] = df.groupby('customer_id').revenue.transform(
    lambda s: s.rolling(7).mean())

# WRONG: centred windows use the future
df['worse'] = df.revenue.rolling(7, center=True).mean()

The .shift(1) before .rolling() is what makes the window strictly historical. Without it, the average includes the value you are trying to predict.

Recency features are usually strong:

snapshot = pd.Timestamp('2026-03-01')       # the prediction date
df['days_since_last_order'] = (snapshot - df.last_order).dt.days
df['days_since_signup'] = (snapshot - df.signup_date).dt.days

Fix the snapshot date explicitly. Using Timestamp.now() makes your features change every time the code runs, which destroys reproducibility and quietly leaks future information into a backtest.

As-of joins for point-in-time attributes

Attaching a customer’s current segment to a two-year-old event is leakage: the segment may have changed because of what happened.

events = events.sort_values('timestamp')
segments = segments.sort_values('valid_from')

merged = pd.merge_asof(events, segments,
                       left_on='timestamp', right_on='valid_from',
                       by='customer_id', direction='backward')

That gives each event the segment as it was at the time. It is more work and it is the difference between a backtest you can trust and one you cannot.

Text Features

In increasing order of complexity, and the simple ones win more often than you expect.

# Cheap and often surprisingly predictive
df['desc_length'] = df.description.str.len()
df['n_words'] = df.description.str.split().str.len()
df['has_number'] = df.description.str.contains(r'\d', na=False).astype(int)
df['all_caps'] = df.description.str.isupper().fillna(False).astype(int)

Then bag of words and TF-IDF:

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(max_features=5000, ngram_range=(1, 2),
                        min_df=5, stop_words='english')

TF-IDF weights words by how distinctive they are, downweighting terms that appear everywhere. min_df=5 drops words seen fewer than five times, which removes typos and enormous numbers of useless columns.

Embeddings from a pre-trained model come last. They are more powerful and much heavier, and on a well-defined classification task with a few thousand examples TF-IDF is frequently competitive.

Missingness as Signal

Worth its own section because it is so often the most predictive thing available.

for col in ['income', 'phone', 'company']:
    df[f'{col}_missing'] = df[col].isna().astype(int)

A blank income field on a credit application, an omitted phone number, an unfilled optional field. These carry real information about the person, and imputing the gap without keeping the flag throws it away.

Interactions

Sometimes the combination matters more than either part.

df['tier_x_channel'] = df.tier.astype(str) + '_' + df.channel.astype(str)
df['price_per_unit'] = df.price / df.quantity

Tree ensembles discover interactions on their own, given enough depth and data. Linear models cannot, so explicit interactions matter far more there.

Resist automated interaction generation across every pair of columns. It produces an enormous number of features, most of them noise, and it makes leakage harder to spot.

Feature Selection

Removing features helps less than people expect, but it does help with speed, cost and explainability.

# 1. Drop constants — no information
constant = [c for c in X.columns if X[c].nunique() <= 1]

# 2. Drop near-duplicates
corr = X.corr(numeric_only=True).abs()
upper = corr.where(np.triu(np.ones(corr.shape), k=1).astype(bool))
redundant = [c for c in upper.columns if (upper[c] > 0.95).any()]

# 3. Then use model-based importance, INSIDE cross-validation
from sklearn.feature_selection import SelectFromModel

That last point matters. Selecting features by their relationship to the target, across the whole dataset, before cross-validating, is leakage. It must happen inside the pipeline.

Prefer permutation importance over a model’s built-in importances, which are biased toward high-cardinality features:

from sklearn.inspection import permutation_importance

r = permutation_importance(model, X_val, y_val, n_repeats=10, random_state=0)
pd.Series(r.importances_mean, index=X_val.columns).sort_values(ascending=False)

It measures what actually degrades when you shuffle a column, which is the question you meant to ask.

Training-Serving Skew

The failure mode that ends otherwise good projects. Features are computed one way in training, using pandas over a historical table, and another way in production, using application code over a live database. The two drift apart, and the model receives inputs it never saw.

Three defences:

  • One implementation. The same function computes features in both places. Not two implementations that are supposed to agree.
  • Log what production actually sent, then compare its distribution against training.
  • Assert on ranges at serving time. If a feature arrives outside the range seen in training, that is worth an alert rather than a silent prediction.

What Carries Forward

Ask the production question about every feature. When is it computed, from where, and is it available in time?

Group-relative features are the highest-return family in business data. transform makes them one line.

Shift before rolling. A window that includes the current row is leakage.

Keep missingness flags. A gap is often the most informative thing on the form.

Compute features once, in one place, used by both training and serving.

Next: getting all of this out of a notebook and into something that runs on a schedule.