pandas missing values: Check before choosing dropna or fillna

Seeing NaN in a DataFrame can make dropna() look like an easy cleanup. But an order with a valid amount may disappear just because an optional note is blank. Filling every gap with fillna() can be equally misleading if a replacement starts to look like a value that was actually observed.

A missing value raises two questions: why is it absent, and what decision uses this column? Inspect the location, rate, and business meaning before deleting or replacing anything.

First inspect missing-value locations and rates with isna

The diagram selects orders 102 and 103 from the larger example below. dropna(subset=['amount']) removes 102 because its amount is missing and retains 103. Filling only memo does not fill 102's amount; the replacement memo string is illustrative.

Here amount is required for the analysis and memo is optional. We use English sample notes without changing which cells are missing:

import pandas as pd

orders = pd.DataFrame(
    {
        "order_id": [101, 102, 103, 104],
        "amount": [12000, None, 18000, 21000],
        "memo": ["At door", None, None, "At reception"],
    }
)

missing_count = orders.isna().sum()
missing_rate = orders.isna().mean()

assert missing_count.to_dict() == {
    "order_id": 0,
    "amount": 1,
    "memo": 2,
}
assert missing_rate["amount"] == 0.25
assert missing_rate["memo"] == 0.5

isna() returns a Boolean for each cell. Summing those Booleans counts missing cells per column; averaging them gives proportions because True counts as one. Do not choose a deletion threshold from a percentage alone. The recovery cost of 25% missing in four rows differs from 25% in millions of rows, and column importance differs too.

Specify which columns dropna is allowed to require

With no arguments, dropna() removes any row missing any column:

all_columns_required = orders.dropna()

assert all_columns_required["order_id"].tolist() == [101, 104]

Order 103 had an amount but vanished because its optional memo was blank. If the analysis requires only an amount, name that column:

amount_required = orders.dropna(subset=["amount"])

assert amount_required["order_id"].tolist() == [101, 103, 104]
assert amount_required["amount"].sum() == 51000

The original orders is unchanged. Assign the result to a new variable or deliberately reassign it so the next stage uses the transformed data. Keeping original and transformed values available makes the decision easier to inspect than an in-place change.

Preserve the distinction between unknown and zero with fillna

An optional note can receive a display placeholder:

display_orders = orders.assign(
    memo=orders["memo"].fillna("No memo")
)

assert display_orders.loc[1, "memo"] == "No memo"
assert display_orders["amount"].isna().sum() == 1

This leaves amount untouched. Filling an unknown amount with 0 could falsely create a free order. Mean or median imputation also changes distributions and may introduce bias; record why a replacement is valid and consider retaining a missingness indicator.

Putting a string such as "Unknown" into a datetime column can change its dtype and disrupt date calculations or ordering. Retain NaT or represent the status in a separate column when that preserves meaning better.

Test the preprocessing contract, not just the row count

def prepare_orders(frame: pd.DataFrame) -> pd.DataFrame:
    required = {"order_id", "amount", "memo"}
    missing_columns = required - set(frame.columns)
    if missing_columns:
        raise ValueError(f"Missing required columns: {sorted(missing_columns)}")

    result = frame.dropna(subset=["amount"]).copy()
    result["memo"] = result["memo"].fillna("No memo")
    return result

prepared = prepare_orders(orders)

assert len(prepared) == 3
assert prepared["order_id"].is_unique
assert prepared["amount"].notna().all()
assert prepared["memo"].notna().all()

If the output surprises you, check input columns and dtypes with info(), inspect isna().sum() and isna().mean(), compare removed IDs, and recheck dtypes and summary statistics after filling. Finally, ask whether missing values concentrate in one time period, sensor, or user group. If missingness is systematic, dropping rows can bias the result.

Watch missingness in a running pipeline

A rule that fit last month's input may silently discard more rows after data collection changes. Record input counts, missing counts by column, removed rows, and replaced cells for each batch. Keep source data and the rule version so you can reproduce or reverse a bad transformation. For sensitive columns, log counts and rates rather than raw names or contact details.

Key takeaways

Use isna() to see where and how often values are absent before choosing dropna or fillna. Limit deletion with subset, and ensure a replacement does not change the meaning or dtype of the data. Test IDs, row counts, dtypes, and summary statistics, then monitor missingness over time.