Pandas — Data opschonen - Machine Learning Cursussen

Ruwe data is zelden klaar voor analyse. Vóór je ook maar één visualisatie maakt of statistiek berekent, moet je de data opschonen. Deze notebook behandelt de meest voorkomende stappen:

Ontbrekende waarden detecteren en behandelen
Kolommen hernoemen en verwijderen
Types corrigeren (astype, to_datetime)
Duplicaten opsporen en verwijderen
Strings opschonen

import numpy as np
import pandas as pd
import seaborn as sns

titanic = sns.load_dataset("titanic")
titanic.head()

Ontbrekende waarden¶

Detectie¶

# Number of NaNs per column
print(titanic.isnull().sum())
print()

# Percentage NaN per column
pct = (titanic.isnull().mean() * 100).round(1)
print(pct[pct > 0])

Strategie 1: Rijen/kolommen verwijderen¶

df = titanic.copy()

# Drop rows with at least one NaN
df_clean = df.dropna()
print(f"Before: {len(df)}, after dropna: {len(df_clean)}")

# Drop rows with NaN in a specific column
df_clean2 = df.dropna(subset=["age"])
print(f"After dropna on 'age': {len(df_clean2)}")

# Drop columns with more than 50% NaN
threshold = 0.5
df_no_sparse = df.loc[:, df.isnull().mean() < threshold]
print("Columns dropped:", set(df.columns) - set(df_no_sparse.columns))

Strategie 2: Opvullen (imputeren)¶

De keuze van opvulstrategie hangt af van het datatype en de distributie:

Numerisch, symmetrisch → gemiddelde
Numerisch, scheef → mediaan
Categorisch → modus (meest voorkomende waarde)

df = titanic.copy()

# Numeric: fill with median
median_age = df["age"].median()
df["age"] = df["age"].fillna(median_age)
print(f"Median age: {median_age:.1f}")
print(f"NaNs in 'age' after filling: {df['age'].isnull().sum()}")

# Categorical: fill with mode
mode_embarked = df["embarked"].mode()[0]
df["embarked"] = df["embarked"].fillna(mode_embarked)
print(f"Mode embarked: {mode_embarked!r}")

Kolommen hernoemen en verwijderen¶

df = titanic.copy()

# Rename with .rename()
df = df.rename(
    columns={
        "survived": "is_survived",
        "pclass": "passenger_class",
        "sibsp": "siblings_spouses",
        "parch": "parents_children",
    }
)
print(df.columns.tolist())

# Drop columns with .drop()
df = df.drop(columns=["deck", "embark_town", "alive", "who"])
print(df.columns.tolist())

Datatypes corrigeren¶

`astype()` — type omzetten¶

df = titanic.copy()

# survived is already 0/1 int, cast to bool
df["survived"] = df["survived"].astype(bool)
print(df["survived"].dtype)

# Category type — more efficient for repeated string values
df["sex"] = df["sex"].astype("category")
df["pclass"] = df["pclass"].astype("category")
print(df[["sex", "pclass"]].dtypes)

`pd.to_datetime()` — datums parseren¶

# Example with fictional date strings
df_dates = pd.DataFrame(
    {
        "date": ["2024-01-15", "2024-03-22", "2024-07-04", "2024-12-31"],
        "value": [100, 150, 130, 180],
    }
)

df_dates["date"] = pd.to_datetime(df_dates["date"])
print(df_dates.dtypes)
print(df_dates["date"].head())

# Extract date components via the .dt accessor
df_dates["year"] = df_dates["date"].dt.year
df_dates["month"] = df_dates["date"].dt.month
df_dates["day_name"] = df_dates["date"].dt.day_name()
df_dates

Duplicaten¶

# Create a DataFrame with duplicates
df_dup = pd.DataFrame(
    {
        "name": ["Alice", "Bob", "Alice", "Charlie", "Bob"],
        "score": [88, 74, 88, 95, 74],
    }
)

print("Duplicates:")
print(df_dup.duplicated())
print(f"\nNumber of duplicates: {df_dup.duplicated().sum()}")

# Remove duplicates (keeps the first occurrence)
df_unique = df_dup.drop_duplicates()
print(df_unique)

Strings opkuisen via `.str` accessor¶

df_str = pd.DataFrame(
    {
        "column_name": ["  Name Student ", "Age", " Score (%) ", "  Major"],
    }
)

# Strip whitespace + lowercase + spaces/special chars → underscore
df_str["clean"] = (
    df_str["column_name"]
    .str.strip()
    .str.lower()
    .str.replace(r"[^a-z0-9]+", "_", regex=True)
    .str.strip("_")
)
df_str

Oefeningen¶

Gebruik de Penguins-dataset: sns.load_dataset("penguins").

penguins = sns.load_dataset("penguins")
penguins.head()

Oefening 1 — Maak een kopie van de dataset (df = penguins.copy()). Bepaal welke kolommen ontbrekende waarden hebben en hoeveel. Verwijder vervolgens alle rijen met een NaN in bill_length_mm of bill_depth_mm.

df = penguins.copy()
# your solution here

Oefening 2 — Vul de ontbrekende waarden in body_mass_g op met de mediaan per species (gebruik een groupby + transform). Controleer daarna dat er geen NaN’s meer zijn in die kolom.

df = penguins.copy()
# your solution here
# Tip: df["body_mass_g"] = df.groupby("species")["body_mass_g"].transform(lambda x: x.fillna(x.median()))

Oefening 3 — Hernoem de kolommen bill_length_mm, bill_depth_mm, flipper_length_mm en body_mass_g naar respectievelijk bill_length, bill_depth, flipper_length en body_mass.

df = penguins.copy()
# your solution here

Oefening 4 — Zet de kolom species om naar het Pandas category dtype. Zet body_mass_g om naar float32 in plaats van float64. Controleer de dtypes na.

df = penguins.copy()
# your solution here

Oefening 5 — Maak een DataFrame met fictieve data die 3 duplicaten bevat. Detecteer de duplicaten, druk ze af, en verwijder ze.

# your solution here

Pandas — Data opschonen