Natural experiments

Natural Experiments - Nature's Own RCT

• An observational study where an external event (e.g., policy change, natural disaster) creates quasi-random assignment to 'treatment' and 'control' groups.
• Leverages naturally occurring phenomena to mimic an RCT without direct investigator intervention.

Key Characteristics:

• Strength: Allows study of exposures that are unethical or impractical to randomize (e.g., effects of a famine, pollution).
• Weakness: High risk of confounding; groups may not be truly comparable at baseline as assignment is not perfectly random.

⭐ Classic Example: John Snow’s 1854 cholera study, linking contaminated water from the Broad Street pump to cholera deaths, is a foundational natural experiment.

Key Features - The 'As-If' Random Rule

• Core Principle: A naturally occurring event or policy creates "treatment" and "control" groups. The assignment mechanism is argued to be random or "as-if" random.
• Quasi-Randomization: Exposure is determined by external forces (e.g., a new law, a geographical boundary), not by investigators. This process approximates the randomization of a true RCT.
• Confounder Mitigation: The "as-if" random assignment is assumed to distribute both measured and unmeasured confounders evenly between groups, thus strengthening causal claims over other observational designs.

⭐ The validity of a natural experiment's conclusions hinges entirely on how plausible the "as-if" random assumption is. If the groups differ systematically at baseline, the study suffers from confounding.

Analysis Techniques - Decoding the Data

• Core Goal: Isolate the causal effect of an exposure by comparing outcomes between groups that are quasi-randomly assigned to an intervention.
• Key Challenge: Controlling for confounding variables that differ systematically between the exposed and unexposed groups.

• Difference-in-Differences (DiD)

  • Calculates the treatment effect by comparing the pre-to-post-intervention change in the outcome for the treatment group versus the control group.
  • Formula: $DiD = (Y_{post}^{treat} - Y_{pre}^{treat}) - (Y_{post}^{control} - Y_{pre}^{control})$

• Regression Discontinuity Design (RDD)

  • Compares individuals just above and below a specific treatment eligibility threshold.
  • Assumes subjects near the cutoff are similar in all other aspects.

⭐ The most critical assumption in DiD is parallel trends: the outcome trends in the treatment and control groups would have been identical in the absence of the intervention. Testing this assumption with pre-intervention data is a key step.

Strengths & Weaknesses - Great Power, Great Caveats

• Strengths

  • High external validity (real-world applicability).
  • Allows study of exposures unethical or impractical to randomize (e.g., policy changes, disasters).
  • Mimics randomization, offering stronger causal inference than purely observational designs.
  • Often cost-effective as the intervention occurs naturally.

• Weaknesses

  • Major risk of confounding from unmeasured variables.
  • Exposure isn't truly random, risking selection bias.
  • Difficult to define or quantify the precise "exposure."
  • Relies on rare or unpredictable events.

⭐ The validity of a natural experiment hinges on the "as-if random" assumption. If the naturally exposed and unexposed groups differ systematically on baseline characteristics, causal claims are weakened.

• In a natural experiment, the exposure is not controlled or assigned by the investigator; it occurs due to external, “natural” events.
• These studies leverage naturally occurring events or policy changes that approximate random assignment to different groups.
• They are crucial for evaluating interventions that are unethical or impractical to randomize, like the effects of new laws or disasters.
• A major strength is the potential for strong causal inference, often mimicking an RCT.
• However, they are highly susceptible to unmeasured confounding since true randomization is not controlled by researchers.