Pandas: Manipulate and Analyze Data in Python

Pandas is the go-to library for manipulating tabular data in Python. Built on NumPy, it provides two essential data structures, the Series and the DataFrame, along with a complete toolkit for loading files, cleaning missing values, filtering, aggregating, joining, and exporting data. This guide covers the full workflow of an analysis: from reading a CSV to exporting the results, including the classic pitfalls to avoid.

Series and DataFrame: the two fundamental structures

The Series: a labeled column

A Series is a one-dimensional array in which each value is associated with an index label.

import pandas as pd

s = pd.Series([15, 12, 17, 9], index=["Alice", "Bob", "Chloé", "David"])
print(s)
# Alice    15
# Bob      12
# Chloé    17
# David     9
# dtype: int64

# Access by label or by position
print(s["Chloé"])       # 17
print(s.iloc[0])        # 15

# Operations are vectorized, as with NumPy
print(s.mean())         # 13.25
print(s + 1)            # each grade increased by 1

The DataFrame: a two-dimensional table

A DataFrame is a collection of columns (Series) that share the same row index.

df = pd.DataFrame({
    "nom": ["Alice", "Bob", "Chloé", "David"],
    "age": [25, 32, 28, 41],
    "ville": ["Paris", "Lyon", "Paris", "Nantes"],
    "salaire": [42000, 38000, 51000, 46000],
})
print(df)
#      nom  age   ville  salaire
# 0  Alice   25   Paris    42000
# 1    Bob   32    Lyon    38000
# 2  Chloé   28   Paris    51000
# 3  David   41  Nantes    46000

# A column is a Series
print(df["age"].dtype)    # int64
print(df.shape)           # (4, 4)
print(df.columns.tolist())  # ['nom', 'age', 'ville', 'salaire']

Reading files with read_csv

read_csv is the most widely used Pandas function. Its options cover nearly every CSV format you’ll encounter in practice.

# Simple read
df = pd.read_csv("ventes.csv")

# Common options
df = pd.read_csv(
    "ventes.csv",
    sep=";",                     # separator (semicolon is common in France)
    decimal=",",                 # decimal comma
    encoding="utf-8",            # or "latin-1" for older files
    usecols=["date", "produit", "montant"],   # load only certain columns
    dtype={"produit": "string"}, # force a type
    parse_dates=["date"],        # convert to datetime at read time
    na_values=["N/A", "inconnu"],  # values to treat as missing
    nrows=1000,                  # limit the number of rows (useful for exploring)
)

Pandas also reads many other formats: read_excel, read_json, read_parquet, read_sql. For large datasets, the Parquet format is significantly faster and more compact than CSV.

Inspection: head, info, describe

The first three commands to run on any new dataset:

# Preview the first rows
print(df.head())        # first 5 rows (df.tail() for the last ones)

# Structure: types, non-null values, memory
df.info()
# <class 'pandas.core.frame.DataFrame'>
# RangeIndex: 4 entries, 0 to 3
# Data columns (total 4 columns):
#  #   Column   Non-Null Count  Dtype
# ---  ------   --------------  -----
#  0   nom      4 non-null      object
#  1   age      4 non-null      int64
#  2   ville    4 non-null      object
#  3   salaire  4 non-null      int64

# Descriptive statistics for numeric columns
print(df.describe())
#              age       salaire
# count   4.000000      4.000000
# mean   31.500000  44250.000000
# std     6.952218   5560.275773
# min    25.000000  38000.000000
# max    41.000000  51000.000000

# Count distinct values in a column
print(df["ville"].value_counts())
# Paris     2
# Lyon      1
# Nantes    1

Selection with loc and iloc

Two complementary accessors: loc selects by labels, iloc by integer positions.

# loc[rows, columns] by labels
print(df.loc[0, "nom"])                  # Alice
print(df.loc[0:2, ["nom", "ville"]])     # rows 0 to 2 INCLUSIVE, two columns

# iloc[rows, columns] by positions
print(df.iloc[0, 0])                     # Alice
print(df.iloc[0:2, 0:2])                 # rows 0 and 1 (upper bound excluded), columns 0 and 1

# All rows, one column
print(df.loc[:, "salaire"])

# Last row
print(df.iloc[-1])

Watch out for the difference in convention: loc[0:2] includes the upper bound 2 (label-based logic), whereas iloc[0:2] excludes it (position-based logic, like Python lists).

Boolean filtering

# A single condition
seniors = df[df["age"] > 30]
print(seniors)
#      nom  age   ville  salaire
# 1    Bob   32    Lyon    38000
# 3  David   41  Nantes    46000

# Combined conditions: & (and), | (or), ~ (not), parentheses required
cible = df[(df["ville"] == "Paris") & (df["salaire"] >= 45000)]
print(cible)
#      nom  age  ville  salaire
# 2  Chloé   28  Paris    51000

# Membership in a list
grandes_villes = df[df["ville"].isin(["Paris", "Lyon"])]

# Filtering on text
noms_a = df[df["nom"].str.startswith("A")]

# query: a readable alternative syntax
resultat = df.query("age > 30 and salaire < 47000")

Computed columns

# New column from existing ones
df["salaire_mensuel"] = df["salaire"] / 12

# Conditional column with numpy.where
import numpy as np
df["categorie"] = np.where(df["age"] < 30, "junior", "senior")

# Transformation with a function (reserve for non-vectorizable cases)
df["initiale"] = df["nom"].apply(lambda n: n[0])

# assign: create columns while chaining operations
df2 = df.assign(
    salaire_k=df["salaire"] / 1000,
    age_en_2030=df["age"] + 4,
)
print(df2[["nom", "salaire_k", "age_en_2030"]].head(2))
#      nom  salaire_k  age_en_2030
# 0  Alice       42.0           29
# 1    Bob       38.0           36

Always favor vectorized operations (df["a"] + df["b"], np.where, .str, .dt): apply with a lambda is often 10 to 100 times slower.

Missing values: isna, fillna, dropna

df_nan = pd.DataFrame({
    "produit": ["A", "B", "C", "D"],
    "prix": [10.0, None, 25.0, None],
    "stock": [5, 3, None, 8],
})

# Detect missing values
print(df_nan.isna())
print(df_nan.isna().sum())     # number of NaN per column
# produit    0
# prix       2
# stock      1

# Replace with a fixed value
df_rempli = df_nan.fillna({"prix": 0.0, "stock": 0})

# Replace with a statistic
df_nan["prix"] = df_nan["prix"].fillna(df_nan["prix"].mean())
print(df_nan["prix"])
# 0    10.0
# 1    17.5
# 2    25.0
# 3    17.5

# Drop rows containing at least one NaN
df_propre = df_nan.dropna()

# Drop only if a specific column is missing
df_propre = df_nan.dropna(subset=["stock"])

Choose your strategy based on context: dropping rows loses information, and imputing a mean can bias an analysis. Always document your choice.

groupby and aggregations

The “split-apply-combine” pattern: split the data into groups, apply a calculation, and recombine the results.

ventes = pd.DataFrame({
    "region": ["Nord", "Sud", "Nord", "Sud", "Nord"],
    "produit": ["A", "A", "B", "B", "A"],
    "montant": [100, 150, 200, 120, 80],
})

# Simple aggregation
print(ventes.groupby("region")["montant"].sum())
# region
# Nord    380
# Sud     270

# Several statistics at once
print(ventes.groupby("region")["montant"].agg(["sum", "mean", "count"]))
#         sum        mean  count
# region
# Nord    380  126.666667      3
# Sud     270  135.000000      2

# Grouping on multiple columns
print(ventes.groupby(["region", "produit"])["montant"].sum())
# region  produit
# Nord    A          180
#         B          200
# Sud     A          150
#         B          120

# Named aggregations: explicit output columns
resume = ventes.groupby("region").agg(
    total=("montant", "sum"),
    panier_moyen=("montant", "mean"),
)
print(resume)
#         total  panier_moyen
# region
# Nord      380    126.666667
# Sud       270    135.000000

To turn a grouped result into a cross-tabulation, pivot_table is often more direct:

tableau = ventes.pivot_table(index="region", columns="produit",
                             values="montant", aggfunc="sum", fill_value=0)
print(tableau)
# produit    A    B
# region
# Nord     180  200
# Sud      150  120

Combining tables: merge and concat

merge: SQL-style joins

clients = pd.DataFrame({
    "client_id": [1, 2, 3],
    "nom": ["Alice", "Bob", "Chloé"],
})
commandes = pd.DataFrame({
    "client_id": [1, 1, 2, 4],
    "montant": [50, 30, 70, 20],
})

# Inner join: matches only
interne = pd.merge(clients, commandes, on="client_id", how="inner")
print(interne)
#    client_id    nom  montant
# 0          1  Alice       50
# 1          1  Alice       30
# 2          2    Bob       70

# Left join: all customers, even those without an order
gauche = pd.merge(clients, commandes, on="client_id", how="left")
# Chloé appears with montant NaN

# how also accepts "right" and "outer"

concat: stacking tables

janvier = pd.DataFrame({"produit": ["A", "B"], "ventes": [10, 20]})
fevrier = pd.DataFrame({"produit": ["A", "B"], "ventes": [15, 25]})

# Vertical stacking with reindexing
annee = pd.concat([janvier, fevrier], ignore_index=True)
print(annee)
#   produit  ventes
# 0       A      10
# 1       B      20
# 2       A      15
# 3       B      25

Sorting

# Sort on a single column
print(df.sort_values("age"))

# Descending sort on multiple columns
print(df.sort_values(["ville", "salaire"], ascending=[True, False]))

# Sort on the index
print(df.sort_index())

# The n largest values (faster than a full sort)
print(df.nlargest(2, "salaire")[["nom", "salaire"]])
#      nom  salaire
# 2  Chloé    51000
# 3  David    46000

Dates: to_datetime and resample

journal = pd.DataFrame({
    "date": ["2026-01-05", "2026-01-12", "2026-02-03", "2026-02-20", "2026-03-01"],
    "ca": [120, 80, 200, 150, 90],
})

# Convert to datetime
journal["date"] = pd.to_datetime(journal["date"])
print(journal["date"].dtype)     # datetime64[ns]

# The .dt accessor exposes the components
journal["mois"] = journal["date"].dt.month
journal["jour_semaine"] = journal["date"].dt.day_name()

# resample: time-based aggregation (requires a datetime index)
serie = journal.set_index("date")["ca"]
mensuel = serie.resample("ME").sum()     # "ME" = month end (pandas >= 2.2)
print(mensuel)
# date
# 2026-01-31    200
# 2026-02-28    350
# 2026-03-31     90

# Filter a date range
recentes = journal[journal["date"] >= "2026-02-01"]

If the format is non-standard, specify it: pd.to_datetime(s, format="%d/%m/%Y") avoids day/month ambiguity.

Exporting results

# CSV (index=False to avoid writing the index column)
df.to_csv("resultats.csv", index=False, sep=";", encoding="utf-8")

# Excel (requires openpyxl)
df.to_excel("resultats.xlsx", sheet_name="Analyse", index=False)

# Parquet: fast, compact, types preserved (requires pyarrow)
df.to_parquet("resultats.parquet")

# JSON
df.to_json("resultats.json", orient="records", force_ascii=False)

Common pitfalls: the SettingWithCopyWarning

This warning appears when you modify the result of a selection for which Pandas can’t tell whether it is a view or a copy.

# BAD: chained indexing, the modification may be lost
sous_ensemble = df[df["ville"] == "Paris"]
sous_ensemble["salaire"] = 0        # SettingWithCopyWarning

# GOOD: modify the original DataFrame in a single loc operation
df.loc[df["ville"] == "Paris", "salaire"] = 0

# GOOD: work on an explicit, intentional copy
sous_ensemble = df[df["ville"] == "Paris"].copy()
sous_ensemble["salaire"] = 0        # no warning, original df untouched

Other frequent pitfalls:

• Comparing with NaN: df["col"] == np.nan always returns False; use df["col"].isna().
• Silent object types: a column of numbers containing a string becomes object; check with df.dtypes and convert with pd.to_numeric(s, errors="coerce").
• inplace=True: discouraged by the Pandas team; prefer reassignment: df = df.dropna().

Best practices

• Always inspect every new dataset with head, info, and describe before any transformation.
• Use loc/iloc explicitly for any write operation, and avoid chained indexing (df[...][...] = ...).
• Vectorize: avoid apply and loops when a whole-column operation exists.
• Parse dates at read time with parse_dates rather than converting afterward.
• Name your aggregations with agg(total=("montant", "sum")) to get clear output columns.
• Prefer Parquet over CSV for intermediate files: faster, more compact, and with no loss of types.
• Chain transformations readably with assign, query, and pipe instead of multiplying intermediate variables.

Conclusion

Pandas covers the entire lifecycle of a data analysis: robust loading with read_csv, exploration with info and describe, cleaning of missing values, boolean filtering, computed columns, aggregations with groupby, joins with merge, fine-grained date handling, and multi-format export. By understanding the difference between loc and iloc and avoiding chained indexing, you eliminate the vast majority of subtle bugs. The natural next steps are visualizing your DataFrames with Matplotlib and performing statistical analysis with SciPy, two libraries that integrate natively with Pandas.