Shortcut to Sections

Descriptive Statistics

Hypothesis Testing

A/B Testing

Probability Distributions

Probability

Misc Terms

Bias


What to cover on a "how would you test __" question / opportunity to talk about what to do with some data (in code with assumption that you use coderpad to outline response)

## DISTRIBUTION
	# How: poisson / log-normal / etc
	# Why: explain (count over time / minimum of 0 / etc)
	# Impact: probability of surge, expected lift for A/B test, segment identification

## VALIDATE A/B TESTING NEED
	# Need: intuition about why there would be lift
	# CLT: statistically allows us to perform a t- or z-test

## ACTUAL TESTING
	# Metric: 
	# Thresholds:
		# alpha = 0.05 / num_tests
		# power = 0.8
		# MDE = gather from domain knowledge + stakeholders
		# Duration / sample size = calculate from MDE, alpha, power

## INTERPRET + LAUNCH
	# Evaluate:
		# Significance: alpha?
		# Size / Impact: effect size, actually impactful to users?
		# Effort: how hard is the change to implement?
		# Risk: how were guardrails effected? other potential downsides?
	# Launch: use a small holdout group for longer-term analysis


Descriptive Statistics

Back to top

Measures of Central Tendency