P-Value Explained Simply: What 0.05 Actually Means (And Doesn’t)

P-Value Explained Simply: What 0.05 Actually Means (And Doesn’t)

Every week someone sends me a study with a highlighted p-value and the message: “See? It’s significant!” And every week I have to explain that significance doesn’t mean what they think it means. After fifteen years of teaching statistics and living with a brain that refuses to memorize formulas without understanding the logic behind them, I’ve learned one thing — the p-value is simultaneously the most used and most misunderstood number in all of science.

Related: evidence-based teaching guide

If you work with data, read research reports, or sit in meetings where someone waves around a bar chart, this explanation is for you. We’re going to build a real understanding of p-values from the ground up, without drowning in Greek letters.

Start With the Question Statistics Is Actually Asking

Before we touch the number 0.05, we need to back up and understand what problem statistics is trying to solve. You run an experiment. You get results. But here’s the uncomfortable truth: even if your treatment does absolutely nothing, you will almost always see some difference between your groups just because of random chance.

Flip a fair coin ten times and you might get seven heads. That doesn’t mean the coin is rigged — it means randomness is noisy. The core challenge in statistics is figuring out: is what I’m seeing a real signal, or is it the kind of noise I’d expect even if nothing interesting is happening?

This is where the null hypothesis enters. The null hypothesis is the boring baseline — the assumption that there’s no effect, no difference, no relationship. It’s essentially saying: “Your treatment did nothing. Any difference you see is just random variation.” The p-value is calculated under this assumption.

What a P-Value Actually Is

Here’s the precise definition, and I want you to read it slowly: the p-value is the probability of getting results at least as extreme as the ones you observed, assuming the null hypothesis is true.

Let that sit for a second. The p-value is not asking “is my hypothesis true?” It’s asking a much stranger question: “If there were genuinely no effect, how often would I stumble onto data this surprising just by chance?”

A small p-value — say, 0.02 — means: if the null hypothesis were true, there’s only a 2% chance of getting data this extreme. That’s suspicious. It makes you doubt the null hypothesis. A large p-value — say, 0.40 — means: even if the null hypothesis is true, results like these would happen 40% of the time. Nothing suspicious here.

So when researchers set a threshold of p < 0.05, they’re saying: “I will doubt the null hypothesis when the probability of seeing this data by chance is less than 5%.” That 5% cutoff — one in twenty — became the standard largely because of statistician Ronald Fisher, who suggested it as a convenient rule of thumb in the 1920s. It was never meant to be a universal law (Wasserstein & Lazar, 2016).

The Coin Flip Example That Makes This Concrete

Let’s make this viscerally real. Suppose I claim I have a magic ability to predict coin flips. You test me. We flip a coin 20 times and I get 15 right.

The null hypothesis: I have no ability. I’m just guessing. Under that assumption, the probability of getting 15 or more correct out of 20 by pure luck is about 2.1%. That’s your p-value: roughly 0.021.

Since 0.021 < 0.05, most researchers would say this result is “statistically significant.” They would reject the null hypothesis. But notice what that means carefully — it doesn’t prove I have psychic powers. It says: if I were just guessing, results this good would only happen about 2% of the time. It makes the “just guessing” explanation look unlikely.

Now imagine we only flip the coin 5 times and I get 4 right. The probability of that happening by chance is about 19%. p = 0.19. Not significant. Does that mean I have no ability? No — it might just mean 5 flips is not enough data to detect a real but modest ability. This distinction matters enormously.

The Four Things P-Values Are NOT

This is where most confusion lives. Let me be direct about what a p-value does not tell you, because these misconceptions show up in boardrooms, newsrooms, and unfortunately, peer-reviewed journals.

1. It Is Not the Probability That Your Results Are Due to Chance

People constantly say “p = 0.03 means there’s only a 3% chance my results are due to chance.” This sounds right but it’s backwards. The p-value assumes the null hypothesis is true and asks how likely your data is. It does not directly tell you the probability that your hypothesis is correct. Confusing these two things is a well-documented logical error called the “transpose conditional” fallacy (Goodman, 2008).

2. It Is Not a Measure of Effect Size

A tiny, trivial effect can produce a fantastically small p-value if your sample size is large enough. Imagine studying whether listening to background music increases typing speed. With 100,000 participants, you might find that music increases speed by 0.3 words per minute — an effect so small it’s operationally meaningless — but your p-value could be 0.0001. Statistically significant, practically irrelevant.

This is why good researchers always report effect sizes (like Cohen’s d or r-squared) alongside p-values. Effect size tells you how big the difference is. The p-value only tells you whether you should take the difference seriously as not being random noise.

3. It Is Not a Measure of Replication Probability

Many scientists mistakenly believe that a p-value of 0.05 means there’s a 95% chance the result would replicate. This is false. The probability that a study with p = 0.05 will replicate is much lower than 95%, often below 50%, depending on the research context (Ioannidis, 2005). The “replication crisis” in psychology and other sciences was partly fueled by this misunderstanding — researchers thought crossing the 0.05 threshold was a reliable signal, and it turned out to be noisier than assumed.

4. It Does Not Tell You Whether Your Study Was Well-Designed

A poorly designed study can produce a statistically significant result. If your measurement tools are biased, if your sample isn’t representative, if your conditions weren’t properly controlled — none of that is captured in the p-value. A small p-value from a bad study is still a result from a bad study. Garbage in, statistically significant garbage out.

Why 0.05 Specifically? And Should We Keep It?

The 0.05 cutoff is essentially historical accident elevated to sacred law. Fisher proposed it as a rough guide. Neyman and Pearson later formalized hypothesis testing with explicit error rates, and 0.05 stuck as a convention across fields that have wildly different needs and stakes (Cohen, 1994).

Think about what 0.05 actually implies at scale. If researchers around the world are testing thousands of hypotheses where the null is actually true, and they all use a 0.05 threshold, then by definition 5% of those tests — one in twenty — will produce a “significant” result purely by chance. With enough researchers testing enough things, false positives will flood the literature.

This gets worse with a phenomenon called p-hacking or “researcher degrees of freedom” — the tendency, often unconscious, to keep collecting data until significance appears, to try multiple analyses and report only the one that worked, or to exclude outliers selectively. These practices can massively inflate false positive rates while still producing an honest-looking p < 0.05 (Wasserstein & Lazar, 2016).

Some fields have responded by moving the threshold. In particle physics, the standard for announcing a discovery is p < 0.000003 — the famous “5 sigma” standard. Genomic studies routinely use p < 0.00000005 to account for millions of simultaneous comparisons. There’s growing momentum in some social sciences to use 0.005 instead of 0.05 as a default threshold. None of these numbers are magic — they all represent a judgment call about how much false positive risk is acceptable given the cost of being wrong.

What Should You Do With This Knowledge?

If you read research — and as a knowledge worker aged 25-45, you almost certainly do — here’s how to engage with p-values more intelligently.

Look for Effect Sizes, Not Just Stars

Many journals denote statistical significance with asterisks (p < 0.05, p < 0.01, p < 0.001). When you see those stars, immediately ask: how big is the actual effect? A study that finds a new training method increases employee productivity by 0.2% might have p = 0.001, but is a 0.2% improvement worth implementing the training? That’s a business question, not a statistics question.

Consider the Prior Plausibility

Bayesian thinking offers a corrective here. Before you see data, how plausible is the hypothesis? A p-value of 0.04 means something very different if you’re testing whether a well-understood drug lowers blood pressure versus whether wearing a lucky bracelet improves exam scores. In the first case, there’s strong prior reason to think the effect is real. In the second, even a significant p-value should be met with skepticism, because unlikely things are more likely to be flukes (Goodman, 2008).

Sample Size Is Not a Nuisance Variable

Small studies can miss real effects (low statistical power). Large studies can make trivial effects look significant. When evaluating any research finding, knowing the sample size is essential for interpreting what a p-value actually means. A study with 50 participants that finds p = 0.04 is much less convincing than a pre-registered study with 2,000 participants finding p = 0.04.

Replications Matter More Than Single Studies

No single p-value, however small, should be treated as definitive. The standard of evidence in science — and in good decision-making — should be based on the convergence of multiple independent studies. If five well-designed studies in different labs all find similar effects, that’s far more informative than one spectacular p-value from a single team (Ioannidis, 2005).

The Honest Summary

The p-value is a useful but limited tool. It answers one specific question — how surprising is this data if there’s truly no effect? — and it answers that question imperfectly, under assumptions that are often only approximately true. It does not tell you whether your hypothesis is correct, how large or meaningful an effect is, or whether your study will replicate.

The number 0.05 is a convention, not a fact about the universe. Different fields use different thresholds for good reasons related to their specific costs of false positives versus false negatives. A clinical trial for a cancer drug has different stakes than a marketing A/B test, and the threshold for “convincing” should reflect those stakes.

What makes someone statistically literate isn’t memorizing that p < 0.05 means significant. It’s understanding that statistical significance is one piece of evidence among several — effect size, study design, replication, prior plausibility, and sample size all need to be considered together. When you read a headline claiming “scientists prove X causes Y,” the useful question isn’t just “was it significant?” but “how big was the effect, how well was the study designed, and has anyone else found the same thing?”

Asking those questions won’t make you popular at meetings where people want clean answers. But it will make you the person in the room who actually understands what the data can and cannot tell us — and in a world increasingly run by research claims, that’s a genuinely valuable thing to be (Cohen, 1994).

Last updated: 2026-03-31

References

• Habibzadeh, F. (2025). The P Value: What It Is and What It Is Not. PMC. Link
• Shimozono, Y. (2026). What Would Be the Effect of Lowering the Threshold for Statistical Significance from P < 0.05 to P < 0.005 in Foot and Ankle Randomized Controlled Trials?. PubMed. Link
• UCLA Law Library. (n.d.). Working with Quantitative Data: Statistical Significance and the p-value. UCLA Law Library Guides. Link
• JMIR Publications. (n.d.). How should p-values be reported?. JMIR Support. Link

Related Reading

• How to Teach Math Conceptually
• Classroom Behavior Management with Positive Reinforcement
• Homework Research Reveals What Schools Hide [2026]