Performance When It Actually Matters — Python Primer

Before anything else: most analysis code runs once and does not deserve any of this. A script that takes four minutes and runs monthly is finished. Optimising it is a hobby.

This chapter is for the cases where it genuinely matters: a job that no longer fits its window, a dataset that no longer fits in memory, or an interactive tool that has become unusable.

Measure First

Never optimise on intuition. Donald Knuth’s line about premature optimisation being the root of all evil gets quoted to death, usually by people skipping the half that matters: he was arguing for measuring first, not for never optimising. Programmers are famously bad at guessing where time goes, and the bottleneck is routinely somewhere nobody suspected.

import time
from contextlib import contextmanager

@contextmanager
def timed(label):
    start = time.perf_counter()
    try:
        yield
    finally:
        print(f"{label}: {time.perf_counter() - start:.2f}s")

with timed("load"):
    df = pd.read_csv('data.csv')
with timed("clean"):
    df = clean(df)
with timed("features"):
    X = build_features(df)

Three lines around each stage tells you where to look. Very often the answer is “loading the file”, and the whole optimisation problem was a file format problem.

For finer detail:

import cProfile, pstats

cProfile.run('build_features(df)', 'profile.out')
pstats.Stats('profile.out').sort_stats('cumulative').print_stats(20)

In a notebook, %timeit for small snippets and %prun for a whole call.

The Five Usual Causes

Nearly all slow pandas code is one of these.

flowchart TD
    S["It is slow"] --> A["<b>1. A Python-level loop</b><br/>iterrows, apply, for over rows"]
    S --> B["<b>2. An accidental copy</b><br/>each step rebuilds the frame"]
    S --> C["<b>3. Wrong dtype</b><br/>object where a category<br/>or number belongs"]
    S --> D["<b>4. Reading too much</b><br/>all columns, all rows,<br/>from CSV"]
    S --> E["<b>5. Repeated work</b><br/>recomputing the same<br/>thing in a loop"]
    style A fill:#f7e8e8,stroke:#c02020

1. Python-level loops

The big one, by a wide margin.

# Worst: iterrows builds a Series per row
for i, row in df.iterrows():
    df.loc[i, 'total'] = row.price * row.qty

# Better but still a Python loop
df['total'] = df.apply(lambda r: r.price * r.qty, axis=1)

# Correct: one vectorised operation
df['total'] = df.price * df.qty

The third version is typically hundreds of times faster than the first. iterrows is the slowest common operation in pandas and there is nearly always an alternative.

Conditional logic vectorises too:

# Slow
df['band'] = df.score.apply(lambda s: 'high' if s > 80 else 'low')

# Fast
df['band'] = np.where(df.score > 80, 'high', 'low')

# Several conditions
df['band'] = np.select(
    [df.score > 80, df.score > 50],
    ['high', 'medium'],
    default='low',
)

When you genuinely cannot vectorise, itertuples is several times faster than iterrows because it does not build a Series per row.

2. Memory and dtypes

df.memory_usage(deep=True).sort_values(ascending=False) / 1e6

deep=True matters. Without it, object columns report the size of the pointers rather than the strings.

def shrink(df):
    """Downcast numerics and categorise low-cardinality text."""
    out = df.copy()
    for c in out.select_dtypes('integer'):
        out[c] = pd.to_numeric(out[c], downcast='integer')
    for c in out.select_dtypes('float'):
        out[c] = pd.to_numeric(out[c], downcast='float')
    for c in out.select_dtypes(['object', 'string']):
        if out[c].nunique() / len(out) < 0.5:
            out[c] = out[c].astype('category')
    return out

before = df.memory_usage(deep=True).sum() / 1e6
df = shrink(df)
print(f"{before:.0f} MB → {df.memory_usage(deep=True).sum()/1e6:.0f} MB")

Reductions of five to ten times are common, mostly from categorising text. Check the ranges before downcasting integers, since int8 tops out at 127.

3. Reading too much

Often the whole problem.

df = pd.read_csv(path, usecols=['id', 'date', 'revenue'])   # fewer columns
df = pd.read_parquet(path, columns=['id', 'revenue'])       # Parquet skips the rest on disk

Convert to Parquet after the first read. It is typically an order of magnitude faster to load and much smaller, and it preserves types so you stop re-parsing.

4. Repeated work

# Recomputes the group means on every iteration
for tier in df.tier.unique():
    avg = df.groupby('tier').revenue.mean()[tier]
    ...

# Compute once
averages = df.groupby('tier').revenue.mean()
for tier, avg in averages.items():
    ...

Cache expensive intermediate results to disk so a rerun is cheap:

from pathlib import Path

def cached(path, build):
    p = Path(path)
    if p.exists():
        return pd.read_parquet(p)
    result = build()
    result.to_parquet(p)
    return result

features = cached('cache/features.parquet', lambda: build_features(df))

Delete the cache directory when the inputs change. Put it in .gitignore.

5. Chained operations

Each step in a long chain materialises a new frame. On a large dataset that is real cost. Filter early to make everything downstream smaller:

# Filter first — everything after operates on less data
result = (df
    .loc[df.date >= '2026-01-01', ['customer_id', 'revenue', 'tier']]
    .groupby('tier')
    .revenue.sum())

Reducing columns and rows as early as possible is the cheapest optimisation available.

Groupby Speed

# Fast: built-in aggregations run in compiled code
df.groupby('tier').revenue.mean()
df.groupby('tier').agg(total=('revenue', 'sum'), n=('id', 'count'))

# Slow: a lambda runs Python once per group
df.groupby('tier').revenue.apply(lambda s: s.mean())

Use the string names of aggregations wherever one exists. They dispatch to compiled implementations; a lambda cannot.

For categorical group keys, pass observed=True or pandas produces a row for every unused combination, which on several categorical columns explodes.

When to Leave pandas

Honesty is more useful than loyalty. Pandas is single-threaded and holds everything in memory, and there are three good exits.

DuckDB when the operation is really a SQL query, especially over files larger than memory.

import duckdb

result = duckdb.sql("""
    SELECT region, date_trunc('month', created_at) AS month,
           SUM(revenue) AS revenue
    FROM 'data/orders/*.parquet'
    WHERE created_at >= '2026-01-01'
    GROUP BY 1, 2
""").df()

It reads Parquet directly, uses every core, streams data larger than memory, and returns a DataFrame. For aggregation over large files it is frequently an order of magnitude faster with no setup at all.

Polars when you want a DataFrame API that is multi-core and can stream.

import polars as pl

result = (pl.scan_parquet('data/orders/*.parquet')   # lazy — nothing read yet
    .filter(pl.col('created_at') >= pl.date(2026, 1, 1))
    .group_by('region')
    .agg(pl.col('revenue').sum())
    .collect())                                       # now it executes

scan_parquet builds a plan and optimises it before touching data, so filters and column selection push down to the file.

Numba for a numerical loop that genuinely cannot vectorise.

from numba import njit

@njit
def cumulative_with_reset(values, resets):
    out = np.empty(len(values))
    total = 0.0
    for i in range(len(values)):
        total = 0.0 if resets[i] else total + values[i]
        out[i] = total
    return out

Numba compiles that loop to machine code. It works on NumPy arrays and plain loops, not on pandas objects, and the first call pays a compilation cost.

Parallelism, Briefly

Python’s global interpreter lock means threads do not speed up pure Python computation. They do help when waiting on the network or disk.

from concurrent.futures import ThreadPoolExecutor      # for I/O waiting
from concurrent.futures import ProcessPoolExecutor     # for CPU work

with ProcessPoolExecutor() as pool:
    results = list(pool.map(process_file, file_paths))

Processes each get their own interpreter, so they genuinely parallelise, at the cost of copying data between them. That copying often eats the gain.

Before reaching for either: NumPy and pandas already release the lock and use multiple cores for many operations, and DuckDB and Polars are parallel by default. Try those first.

Knowing When to Stop

Ask three questions before optimising anything.

How often does this run? Once a month, leave it.

How long does it take now, and what would be acceptable? Four minutes down to three is not worth an afternoon.

What does the profiler say? If eighty per cent of the time is in reading the file, no amount of clever vectorisation elsewhere will help.

The order of effort, cheapest first: read less data, fix the dtypes, remove the Python loops, cache the expensive intermediates, and only then reach for another tool.

What Carries Forward

Measure before optimising. The bottleneck is rarely where you think.

iterrows is almost never the answer. Vectorise, or use np.select for conditionals.

Read fewer columns and use Parquet. Often the entire problem.

Downcast and categorise. Five to ten times less memory for a few lines.

DuckDB for large aggregations, Polars for a parallel DataFrame API, Numba for an unavoidable loop.

Most code does not need any of this. Optimise what is actually hurting.

That is the end of the book. You now have the language, the numerical stack, the tools to load and clean real data, the statistics to know whether a result is real, the discipline to evaluate a model honestly, and the engineering to put it somewhere it runs.

The next book, Learn ML Algorithms, covers what is inside the models.