Key Takeaways
- A p-value measures the probability of obtaining results at least as extreme as observed, assuming the null hypothesis is true
- If p < alpha, reject the null hypothesis (result is statistically significant)
- Common significance level: alpha = 0.05 (5% chance of Type I error)
- A small p-value does NOT mean the effect is large or practically important
- P-values should be reported alongside effect sizes and confidence intervals
What Is a P-Value? A Complete Explanation
| P-Value | Significance Level | Interpretation |
|---|---|---|
| p < 0.001 | α = 0.001 | Very strong evidence against H₀ |
| p < 0.01 | α = 0.01 | Strong evidence against H₀ |
| p < 0.05 | α = 0.05 | Moderate evidence (conventional threshold) |
| p < 0.10 | α = 0.10 | Weak/marginal evidence |
| p ≥ 0.10 | — | Insufficient evidence to reject H₀ |
| p = 1.0 | — | No evidence whatsoever |
| Test | Use Case | Distribution |
|---|---|---|
| One-sample z-test | Known population σ, large n | Normal (z) |
| One-sample t-test | Unknown σ, small n | Student's t |
| Two-sample t-test | Compare two group means | Student's t |
| Paired t-test | Before/after or matched pairs | Student's t |
| Chi-square test | Categorical data, goodness-of-fit | χ² |
| F-test / ANOVA | Compare 3+ group variances | F-distribution |
| Mann-Whitney U | Non-parametric two-group comparison | Normal approx. |
| Measure | Small | Medium | Large |
|---|---|---|---|
| Cohen's d (t-test) | 0.2 | 0.5 | 0.8 |
| Cohen's f² (regression) | 0.02 | 0.15 | 0.35 |
| η² (ANOVA) | 0.01 | 0.06 | 0.14 |
| r (correlation) | 0.10 | 0.30 | 0.50 |
| Odds ratio | 1.5 | 2.5 | 4.0 |
A p-value (probability value) is the probability of obtaining test results at least as extreme as the results actually observed, under the assumption that the null hypothesis is correct. In simpler terms, it tells you how likely your data would occur by random chance if there were no real effect.
The p-value is a cornerstone of frequentist statistical inference and is used across virtually all scientific disciplines to make decisions about whether observed effects are "real" or could simply be due to random sampling variation.
p-value = P(data | H0 is true)
How to Interpret P-Values
Step-by-Step Interpretation
Set Your Significance Level (Alpha)
Before collecting data, choose your alpha level (commonly 0.05). This is your threshold for "statistical significance" - the maximum p-value you'll accept to reject the null hypothesis.
Calculate Your Test Statistic
Compute your z-score, t-score, chi-square, or F-statistic from your sample data using the appropriate formula for your test.
Find the P-Value
Use our calculator or statistical tables to find the probability associated with your test statistic. This gives you the p-value.
Make Your Decision
Compare p-value to alpha: If p < alpha, reject H0 (statistically significant). If p ≥ alpha, fail to reject H0 (not statistically significant).
Common Significance Levels Explained
| Alpha Level | Confidence Level | Z-Critical (Two-Tailed) | Common Use |
|---|---|---|---|
| 0.10 | 90% | 1.645 | Exploratory research |
| 0.05 | 95% | 1.960 | Standard scientific research |
| 0.01 | 99% | 2.576 | Medical/clinical studies |
| 0.001 | 99.9% | 3.291 | Particle physics, genetics |
Understanding Different Statistical Tests
Z-Test
Use when you know the population standard deviation and have a large sample size (n > 30). The z-test compares sample means to a known population mean.
T-Test
Use when the population standard deviation is unknown (estimated from sample). Common for comparing two groups or testing if a sample differs from a hypothesized mean. Requires degrees of freedom (df = n - 1 for one sample).
Chi-Square Test
Use for categorical data to test independence or goodness-of-fit. Degrees of freedom depend on the number of categories: df = (rows - 1)(columns - 1) for independence tests.
F-Test
Use to compare variances between groups (ANOVA) or test regression models. Requires two degrees of freedom values: df1 (numerator) and df2 (denominator).
Pro Tip: One-Tailed vs Two-Tailed Tests
Use a two-tailed test when you want to detect a difference in either direction (most common). Use a one-tailed test only when you have a specific directional hypothesis established before data collection. One-tailed tests have more power but can miss effects in the opposite direction.
Common P-Value Mistakes to Avoid
Misinterpretations to Avoid
Wrong: "P = 0.03 means there's a 3% chance the null hypothesis is true."
Right: "P = 0.03 means there's a 3% chance of seeing results this extreme if the null hypothesis were true."
- Statistical significance is not practical significance - A tiny effect can be statistically significant with a large enough sample
- p = 0.05 is not a magic threshold - p = 0.049 and p = 0.051 are virtually identical
- Failing to reject H0 is not "accepting" it - Absence of evidence is not evidence of absence
- Don't p-hack - Running multiple tests until you find significance inflates false positive rates
- Report exact p-values - Say "p = 0.023" not just "p < 0.05"
P-Values and Effect Size
P-values tell you whether an effect exists, but not how large it is. Always report effect sizes alongside p-values:
- Cohen's d - For comparing two means (small: 0.2, medium: 0.5, large: 0.8)
- Pearson's r - For correlations (small: 0.1, medium: 0.3, large: 0.5)
- Odds Ratio - For binary outcomes in logistic regression
- Eta-squared - For ANOVA (proportion of variance explained)