Probability is the most important concept in statistics

Statistics Tutorial: Statistical Probability

All inferential statistics like the t-test, ANOVA, linear regression etc. are based upon Probability.

Probability can best be understood with a coin toss example. Suppose someone offers you a wager of 1 dollar on the toss of a coin. If the coin lands tails you win 1 dollar, if it lands heads you lose 1 dollar.

This could be a fake coin so you inspect it and confirm one side is heads and one side is tails, so far so good. But maybe the coin is weighted or biased in some way to be more likely to land heads.

So you say, before I enter into this wager, I want to conduct a small experiment. Let’s take a random sample of 10 tosses of the coin and see how often it lands heads.

Knowing that heads and tails are equally likely with a fair coin, and also knowing that a coin toss is random, you wouldn't necessarily expect 5/5 heads/tails. If your experiment shows 7 heads out of 10 tosses (70%), you'd probably think that is pretty common with a fair coin and you might go ahead with the wager.

On the other hand, if you get 10 heads out of 10 tosses, you’d probably say to yourself, “that could happen” with a fair coin, but it’s not very likely. So, you’d likely turn down the wager.

If the stakes were much higher than 1 dollar, you’d probably want even greater confidence the coin is fair. So you could conduct a larger sample size of coin tosses to get a more reliable estimate of the probability the coin lands heads. If you toss the coin 1,000 times and get 700 heads (70%), you’d probably say, that is very unlikely to happen with a fair coin and you wouldn’t take the bet.

The coin toss example is analogous to a research study. The study participants constitute a random sample, or at least reasonably close to a random sample. Let’s say we are studying a new diet for weight loss. We have one group of people on the standard diet and one group on the new diet.

Let’s say we observe an average 10-pound weight loss in the standard diet group and an average 20-pound weight loss in the experimental diet group. Knowing the two groups could be different due to random chance, we would ask “how likely is it that we would get such a big difference in the average weight loss between the two groups if in fact the new diet was neither better nor worse than the standard diet? (i.e. how likely is it the observed difference was due to an unlucky random sample)”.

If the answer is, it is very "unlikely" the difference could be due to random chance alone, we would be confident in concluding the new diet is better than the standard diet.

Note that we didn’t come right out and say, “ah-hah, the new diet IS better”. Instead, we concluded "there is very strong evidence to suggest the new diet is better". That is just like in the coin toss example, we weren’t absolutely sure the coin was biased based on 7 heads out of 10 tosses, but 10 straight heads out of 10 tosses is very strong evidence that the coin was biased.