 Philosophy
 Sample vs Population
 Types of Error
 Multi Arm Bandits
 Measures
 Robust Statistics
 Other Resources
Philosophy
Frequentist
Bayes
Computer Based
Twenty First Century
Sample vs Population
Inferential Statistics
Inferential statistics use a random sample to draw conclusions about the population. Typically, it is not practical to obtain data from every member of a population. Instead, we collect a random sample from a small proportion of the population. From the sample, statistical procedures can infer the likely properties of the population.
For example, it is impractical to measure the height of every adult woman, but you can measure the heights of a random sample and use that information to make generalizations about the heights of all women. For example, a confidence interval provides a range that the population mean height is likely to fall in.
Confidence Interval
This is the simplest in laymens terms http://www.bandolier.org.uk/painres/download/whatis/What_are_Conf_Inter.pdf
https://blog.minitab.com/blog/adventuresinstatistics2/understandinghypothesistestssignificancelevelsalphaandpvaluesinstatistics https://web.ma.utexas.edu/users/mks/statmistakes/overviewfreqCI.html
PValues
https://statisticsbyjim.com/hypothesistesting/interpretingpvalues/ https://statisticsbyjim.com/hypothesistesting/pvaluetipsfalsepositivesmisleadingresults/
RANT: PValues are confusing because they answer an easy question, not the one you want, and are phrased in super confusing terms. PValues are much like the story of the man looking for his keys under a street lamp. A woman comes to him and asks if she can help find the keys, and where the man lost him. The man answer’s over there. The woman says, over there  well why are you looking under the street light!? The man responds, well because it’s dark over there.
You want to know the probability the metric change is caused by our treatment. Aka not a false positive (Type 1 error).
But the PValue is the probability that for this sample of the population, the metric change could be caused by the control as opposed to treatment. (The effect being caused by the control is called the null hypothesis)
*Along these lines, you can think of P values as probabilities that you can multiply. For example, if two independent studies both have P values of 0.05, you can multiply them to obtain a probability of 0.0025. If you use this approach, you can’t cherry pick the best studies. You need to include all studies in a series of relevant studies whether they are significant or not.
One P value multiplied by another P value equals a smaller P value. You should consider results from a study in conjunction with other similar studies. It is extremely unlikely that a single study can prove that the alternative hypothesis is true with any confidence. So, don’t expect it to!*
How does Amazon’s WebLab tool handle this?
How does Facebook Deltoid tool handle this?
Types of Error
Type 1 vs Type 2
 Type 1 = False Positives
 Type 2 = False Negative
Precision vs Recall
In ML, we talk about precision and recall:

Precision (Exactness)  Likely hood a result is a true positive. TP / (TP + FP). % of things you say are true, are actually true.

Recall (Completeness)  Likely a true positive. TP / (TP + FN). % of true things you identify in the entire space
Accuracy vs Precision
In measurement we talk about accuracy and precision
Imagine a target on a bullseye, and firing multiple bullets:
 Accuracy = How close to the bullseye are the bullets. Arguably this is diffence of median from bullseye
 Precision = How must variance is the “spray” of bullets. Arguably this is the standard deviation.
Multi Arm Bandits
Explore vs Exploit
Episilon Greedy
Soft Max
Measures
Robust Statistics
Other Resources
 Bandit Algorithoms
 Computer Age Statistical Infrence
 Python for Data Analysis
 R CookBook
 Python Data Science Handbook
 Think Stats