How to Read a p-value Without Fooling Yourself
A p-value is not the probability your result is true, and 'not significant' is not 'no effect.' A plain-language guide to what a p-value actually means, the traps, and how to check significance yourself.
The p-value is the most-cited and most-misread number in science. This is a plain-language guide to what it means, what it doesn’t, and how to sanity-check one yourself. If you just want to run the test, use our A/B significance calculator.
What a p-value is
Every significance test starts with a null hypothesis — usually “there is no difference between these two things.” The p-value answers one narrow question:
If the null hypothesis were true, how often would I see data at least this extreme, just by chance?
That’s it. A p-value of 0.03 means: if there were truly no effect, data this striking would show up about 3% of the time. It is a statement about the data under an assumption — P(data | null) — and the direction of that conditional matters enormously.
What a p-value is NOT
- Not the probability the null hypothesis is true. It assumes the null; it cannot also measure it.
- Not the probability your result was a fluke.
- Not a measure of effect size. A microscopic, useless difference can be highly significant with a big enough sample.
- Not a line between “real” and “fake.” p = 0.049 and p = 0.051 carry near-identical evidence.
The three traps that catch everyone
- “Not significant” read as “no effect.” Failing to reject the null is not proving it. Often it just means the sample was too small. Report the confidence interval — if it’s wide, you simply don’t know yet.
- Peeking / optional stopping. Checking an experiment repeatedly and stopping when it crosses 0.05 massively inflates false positives. Decide your sample size in advance.
- Multiple comparisons. Run 20 tests at α = 0.05 and, on average, one will “reach significance” by pure chance. If you test many things, correct for it.
Always read the effect size
Significance tells you an effect is probably not zero. It does not tell you the effect is big enough to matter. A conversion-rate lift of 0.01% can be statistically significant across millions of users and still be worthless. Read the effect size and its confidence interval alongside every p-value — the interval shows you the range of effects still consistent with your data.
Check it yourself
For a two-variant comparison — an A/B test, a before/after rate — you can compute the p-value in seconds with our significance calculator. It runs a two-proportion z-test, returns the p-value, z-score and a 95% confidence interval on the difference, and warns you when your counts are too small for the math to be trustworthy. For how this newsroom applies these rules to its own reporting, see our methodology.
§ Frequently asked questions
What does a p-value actually mean? +
A p-value is the probability of observing data at least as extreme as yours, assuming the null hypothesis (usually 'no difference') is true. It is P(data | null), not P(null is true | data). A p-value of 0.03 does not mean there's a 3% chance the result is a fluke or a 97% chance it's real.
Does 'not statistically significant' mean there is no effect? +
No. It means you did not gather enough evidence to reject the null at your chosen threshold. That can happen because there truly is no effect — or because your sample was too small to detect a real one. Absence of evidence is not evidence of absence.
Is p < 0.05 a meaningful cutoff? +
It's a convention, not a natural boundary. p = 0.049 and p = 0.051 are almost identical evidence. The 0.05 threshold is a useful default for decision-making but should never be treated as the line between 'true' and 'false.'
Data desk lead · PhD, Statistics
Nadia Krishnan leads the DataBackedNews data desk. She holds a PhD in statistics and spent six years building measurement systems before turning to journalism. She writes about reading evidence honestly — significance, sampling, uncertainty — and reviews the newsroom's most data-heavy reporting.