P-Values

So the null hypothesis says there’s no real trends in your data. Everything you’re seeing in your sample is just noise. And we want to find a way to reject that. If we can reject that we can infer there is a real difference, and it’s likely to continue. So we need to find a way to reject the null hypothesis. How do we do that? That’s what we’re going to do in this lesson. We’re going to reject the null hypothesis using something called a p-value. So what does a p-value? The p-value is the answer to the question, if the null hypothesis was true, how easy would it be to get my result? Let’s apply this.

If the null hypothesis is true, how easy would it be to get my result? For example, if a p-value is 0.02, we’re saying you could only get this result 2% of the time if the null was true. So it’s essentially a measure of how rare or uncommon that result would be if the null was actually true. We can use this– first off we can calculate this– but we can use this to reject the null hypothesis. So let’s see how we might do that. So to reject the null hypothesis essentially we just need to be “next to impossible”– I put that in quotes– to get our result if that null hypothesis is true.

Our result needs to be almost impossible to get with the null true. So if it’s almost impossible to get our result with the null true, and we did get our result, the null must be false. That’s the logic. So in other words, we want our p-value to be small. If our p-value was 0.01, we’re saying there’s only a 1% chance I could have gotten this result if the null was true. But I did get the result, so I can reject the null. That’s the logic that we’re going to use. It doesn’t disprove the null, but it gives some evidence against it. The smaller my p-value, the more evidence I have against the null hypothesis.

Now in standard statistics we often reject the null hypothesis if p is less than 0.05 five or 5%. That’s a pretty weak criteria. Many people would advocate using a stronger threshold, such as 0.01, or even 0.001, but the smaller you get this thing the less likely it is that you could have gotten that result if the null was true. But you did get the result. So we could safely reject the null hypothesis as false.