What to cover on a "how would you test __" question / opportunity to talk about what to do with some data (in code with assumption that you use coderpad to outline response)
## DISTRIBUTION
# How: poisson / log-normal / etc
# Why: explain (count over time / minimum of 0 / etc)
# Impact: probability of surge, expected lift for A/B test, segment identification
## VALIDATE A/B TESTING NEED
# Need: intuition about why there would be lift
# CLT: statistically allows us to perform a t- or z-test
## ACTUAL TESTING
# Metric:
# Thresholds:
# alpha = 0.05 / num_tests
# power = 0.8
# MDE = gather from domain knowledge + stakeholders
# Duration / sample size = calculate from MDE, alpha, power
## INTERPRET + LAUNCH
# Evaluate:
# Significance: alpha?
# Size / Impact: effect size, actually impactful to users?
# Effort: how hard is the change to implement?
# Risk: how were guardrails effected? other potential downsides?
# Launch: use a small holdout group for longer-term analysis