The distributions in the visualization show the theoretical sampling distribution for the null distribution (H 0 ) and the sampling distribution under the alternative hypothesis (H a ). Although this site is not meant as a first introduction to NHST, here is a quick summary of the core concepts. Term Explanation α The conditional probability of incorrectly rejecting H 0 when it actually is true. β The conditional probability of failing to reject H 0 when it is false. Power The complement of β (. 1 - β), this is the probability of correctly rejecting H 0 when it is false. H 0 The null hypothesis, usually stated as the population mean being zero, or that there is no difference. However, it does not have to be stated as a zero or no difference hypothesis. H a The alternative hypothesis, usually stated as the population mean being non-zero or greater then or less than zero. Simply put what we are doing when we perform traditional (frequentist) statistical tests, is that we collect some data and then calculate the probability of observing data at least as extreme as our data given that no effect exists in the population. This conditional probability is the p -value, and if it is smaller then α (usually or ) we claim that our findings are “statistically significant”. Moreover, α is the long-run probability of making a Type I error when H 0 is true. The acceptable Type I error rate is set before running the study, and α should not be confused with the p -value from a single study. Before we collect our data we should perform a power analysis. Usually we specify the minimum effect (say Cohen’s d = ) we are interested in finding, set α to and β to (. 80 % power). The power analysis will tell us how large our sample needs to be to achieve this power. Given this sample size, if we rerun our study many times with new random samples 80 % of the time we will correctly reject the null hypothesis, . we will find that p < α. What Null Hypothesis Significance Testing Does Not Tell Us

- It does not give us the probability that our results are due to chance.
- If we reject H 0 with α = this does not mean that we are 95 % sure that the alternative hypothesis is true.
- Rejecting H 0 with α = does not mean that the probability that we have made a type I error is 5 %.
- A p -value does not tell us that our findings are relevant, clinical significant or of any scientific value whatsoever.
- A small p -value does not tell us our results will replicate.
- A small p -value does not indicate a large treatment effect.
- Failing to reject the null hypothesis is not evidence of it being true.
- If our test has 80 % power and we fail to reject the null hypothesis, then this does not mean that the probability is 20 % that the null is true.
- If our test has 80 % power and we do reject the null hypothesis, then this does not mean that the probability is 80 % that the alternative hypothesis is true.