P-values and confidence intervals

P-Values - Judging The Numbers

• Definition: The probability of obtaining the observed study results, or more extreme results, if the null hypothesis (H₀) were true.
• A small p-value suggests the observed data is unlikely if H₀ is true.
• Alpha (α): The pre-set significance level. It's the threshold for rejecting H₀. Conventionally set at < 0.05.

• If p < α (e.g., p < 0.05):
  • Result is statistically significant.
  • We reject the null hypothesis.
  • There is a <5% chance the result is due to random chance alone.
• If p ≥ α (e.g., p ≥ 0.05):
  • Result is not statistically significant.
  • We fail to reject the null hypothesis.

⭐ A p-value does not indicate the size or clinical importance of an effect. A very large sample size can lead to a small p-value for a trivial effect.

Confidence Intervals - The Range Game

• A Confidence Interval (CI) is a range of values that likely contains the true population parameter (e.g., mean). It quantifies the uncertainty around an estimate.
• Formula: $CI = \text{Point Estimate} \pm \text{Margin of Error}$.
• Interpretation: For a 95% CI, we are 95% confident the true population value falls within that range.
• CI Width & Precision:
  • Wider CI → Less precise estimate.
  • Narrower CI → More precise estimate.
  • Affected by: ↑ sample size → ↓ CI width; ↑ confidence level (99% vs 95%) → ↑ CI width.

• Statistical Significance:
  • For mean differences, if the CI includes 0, the result is NOT statistically significant (p ≥ 0.05).
  • For ratios (Odds Ratio, Relative Risk), if the CI includes 1, the result is NOT statistically significant (p ≥ 0.05).

⭐ If the 95% CI for an intervention does not cross the null value (0 for difference, 1 for ratio), the finding is statistically significant with a p-value < 0.05.

Errors & Power - Dodging Pitfalls

• Type I Error (α): The "false positive." You reject a true null hypothesis (H₀).

  • Probability = α (significance level, e.g., 0.05).
  • 📌 You see an effect that isn't there.

• Type II Error (β): The "false negative." You fail to reject a false null hypothesis (H₀).

  • Probability = β.
  • 📌 You are blind to an effect that is there.

• Power (1 - β): Probability of detecting a true effect (correctly rejecting a false H₀).

  • To ↑ Power: ↑ sample size (n), ↑ effect size, or ↑ α.

⭐ Power is typically set to 80% in clinical trials. This means investigators accept a 20% chance of a Type II error (β = 0.20), a common trade-off for feasibility.

High‑Yield Points - ⚡ Biggest Takeaways

• A p-value is the probability of observing data if the null hypothesis is true.
• If p < 0.05, results are statistically significant, and you reject the null hypothesis.
• A confidence interval (CI) provides a range of plausible values for the true population parameter.
• If a 95% CI for a mean difference excludes 0, or for a ratio excludes 1, the result is significant.
• Narrow CIs indicate high precision, often from a larger sample size.
• The p-value cutoff (α) is the probability of a Type I error.