Reporting standards in medical journals

P-values & CIs - The Dynamic Duo

• P-value: Probability of obtaining observed results (or more extreme) if the null hypothesis is true.
  • Significance threshold: $p < 0.05$.
  • Indicates strength of evidence against the null hypothesis.
• Confidence Interval (CI): Range of plausible values for a population parameter (e.g., mean or odds ratio).
  • A 95% CI implies a 95% probability that this range contains the true value.
  • If the CI for a difference doesn't include 0 (or 1 for a ratio), the result is statistically significant at the corresponding alpha level.

⭐ A confidence interval is superior to a p-value because it conveys both statistical significance and the precision of the effect estimate.

Reporting Pitfalls - Common Journal Gaffes

• P-hacking (Selective Reporting): Reporting only favorable data or analyses that yield significant p-values, creating a biased view of the evidence.
• Misinterpreting Non-Significance: Incorrectly concluding "no effect" or "no association" when a p-value is > 0.05. It only means the observed data are not sufficient to reject the null hypothesis.
• Confusing Statistical vs. Clinical Significance: A small p-value (e.g., p < 0.001) doesn't guarantee a large or clinically meaningful effect. Large sample sizes can make trivial effects statistically significant.
• Omitting Confidence Intervals: Reporting only a p-value without the CI hides the precision and magnitude of the effect estimate. A wide CI indicates high uncertainty.

⭐ Absence of evidence is not evidence of absence. A non-significant p-value does not prove the null hypothesis is true.

The Rulebook - Journal Reporting Standards

• P-values:

  • Report exact values (e.g., p=0.02), not just thresholds (p < 0.05).
  • State the pre-specified significance level (α), usually 0.05.
  • Avoid misinterpreting the p-value as the probability that the null hypothesis is true.

• Confidence Intervals (CIs):

  • Report CIs for all primary effect estimates (e.g., Relative Risk, Odds Ratio).
  • The 95% CI provides a range of plausible values for the true effect and indicates the precision of the estimate.

• Reporting Guidelines:

  • Adhere to CONSORT (CONsolidated Standards of Reporting Trials) for RCTs to ensure transparency and completeness. Adherence is mandated by most major journals.

⭐ If the 95% CI for a ratio (e.g., OR, RR) does not contain the null value of 1.0, the result is statistically significant (p < 0.05).

Good vs. Bad - A Reporting Showdown

| Imprecise statements: $p < 0.05## Good vs. Bad - A Reporting Showdown

| | Provide effect size & 95% CI: e.g., RR 1.5 (95% CI 1.1-2.1) | Isolate p-values: No context of effect size or CI | | Interpret CI: Focus on range of possible effects | Binary thinking: Equating non-significance with "no effect" |

⭐ If the 95% CI for a mean difference contains 0 (or for an odds/risk ratio contains 1), the result is not statistically significant ($p > 0.05$).

• Report exact p-values (e.g., p=0.02) instead of just thresholds (e.g., p<0.05).
• Confidence intervals (CIs) are superior to p-values, showing effect size and precision.
• A statistically significant result has a 95% CI that excludes the null value.
• The null value is 0 for a difference (e.g., mean difference).
• The null value is 1 for a ratio (e.g., odds ratio, relative risk).
• CIs provide the range of plausible values for the true effect.