Intermediate pandas and Statistics Interview Questions

agg returns one row per group; transform returns a value for every original row, aligned with the input. Use transform to compare rows with their group:

import pandas as pd

df = pd.DataFrame({"team": ["A", "A", "B"], "score": [10, 20, 30]})
print(df.groupby("team")["score"].agg("mean").to_dict())
print(df.groupby("team")["score"].transform("mean").tolist())

{'A': 15.0, 'B': 30.0}
[15.0, 15.0, 30.0]

Sort by revenue, then take the first rows of each group:

import pandas as pd

sales = pd.DataFrame({
    "region": ["N", "N", "N", "S", "S", "S"],
    "product": ["Pen", "Book", "Bag", "Pen", "Book", "Bag"],
    "revenue": [300, 900, 450, 200, 150, 800],
})
top = sales.sort_values("revenue", ascending=False).groupby("region").head(2)
print(top.sort_values(["region", "revenue"], ascending=[True, False]))

  region product  revenue
1      N    Book      900
2      N     Bag      450
5      S     Bag      800
3      S     Pen      200

groupby(...).head(n) keeps the first n rows of each group in their current order. An alternative is rank with transform-style grouping and filtering rank <= 2.

import pandas as pd

monthly = pd.Series([120, 150, 135], index=["Jan", "Feb", "Mar"])
print((monthly.pct_change() * 100).round(1).tolist())

[nan, 25.0, -10.0]

With several groups (stores), use groupby("store")["revenue"].pct_change() so growth does not cross from one store into the next.

import pandas as pd

df = pd.DataFrame({"region": ["N", "S", "N", "S"], "month": ["Jan", "Jan", "Feb", "Feb"],
                   "revenue": [100, 80, 120, 90]})
print(pd.pivot_table(df, index="region", columns="month", values="revenue",
                     aggfunc="sum", margins=True, margins_name="Total"))

month   Feb  Jan  Total
region
N       120  100    220
S        90   80    170
Total   210  180    390

Always set aggfunc explicitly: the default is the mean.

iterrows runs Python code once per row and builds a Series for each, which is slow on large data. Prefer vectorised column operations (df["a"] * df["b"]), np.where/np.select for conditions, .str and .dt methods, groupby/merge, and only then apply.

When the data is skewed or has outliers, such as incomes, house prices or delivery times. One very large value pulls the mean up but barely moves the median:

import pandas as pd

salaries = pd.Series([40, 45, 50, 55, 400])
print(salaries.mean(), salaries.median())

118.0 50.0

Plot first (box plot, histogram). Then use a rule: the IQR rule flags values below Q1 − 1.5·IQR or above Q3 + 1.5·IQR and is robust; z-scores beyond ±3 assume roughly normal data and are themselves inflated by the outliers. Investigate before removing: errors should be fixed or removed, real extremes kept or analysed separately.

That they move together strongly, not that one causes the other. A confounder (hot, sunny weather) drives both. Establishing causation needs a controlled experiment (like an A/B test) or careful causal analysis.

"If the change made no difference at all, how often would we see a gap this big just by luck?" A p-value of 0.02 means about 2% of the time. Small p-values mean the result is hard to explain by luck alone. It is not the probability that the change works, and it says nothing about whether the effect is big enough to matter.

A Type I error is a false positive: concluding there is an effect when there is none (its rate is α, often 5%). A Type II error is a false negative: missing a real effect, usually because the sample was too small. Larger samples reduce Type II errors; a lower α reduces Type I errors at the cost of more Type II errors.