Statistical Testing

Statistical testing allows researchers to determine whether the results of a study are due to chance or reflect a true effect. This process revolves around evaluating a hypothesis using mathematical reasoning.

Key Takeaways

• The null hypothesis (H0) assumes no effect; the alternative hypothesis (Ha) proposes one exists.

• A p-value less than the significance level (α), typically 0.05, leads to rejecting H0 as statistically significant.

• A Type I error (false positive) rejects a true H0; a Type II error (false negative) fails to reject a false H0.

• Statistical power — improved by larger sample or effect size — reduces the risk of Type II errors.

• A confidence interval that excludes the null value indicates statistical significance.

Null and Alternative Hypotheses

At the heart of statistical testing lies the null hypothesis (H0) and the alternative hypothesis (Ha).

The null hypothesis is a statement of no effect or no difference — it assumes any observed variation in the data is purely due to chance. In a clinical trial, the null hypothesis might state that a new drug has no effect on blood pressure compared to a placebo.

The alternative hypothesis proposes that there is an effect or a difference. It can be:

• Directional — predicting the direction of the effect (e.g., "the drug lowers blood pressure")

• Non-directional — predicting only that a difference exists, without specifying direction

The goal of statistical testing is to decide whether to reject the null hypothesis in favor of the alternative.

P-Values and Significance Level

To make that decision, researchers calculate a p-value — the probability of observing the results obtained (or something more extreme) if the null hypothesis were true.

The p-value is compared to a predefined significance level (α), often set at 0.05. If the p-value is less than α, the null hypothesis is rejected and the results are considered statistically significant.

MCAT Callout — Worked Example: Interpreting a P-Value: p-value = 0.03 means there is only a 3% chance the observed results occurred under the null hypothesis — leading to rejection of H0.

Type I and Type II Errors

Rejecting or failing to reject H0 comes with the risk of two kinds of error:

Statistical Power

Statistical power is the probability of correctly rejecting the null hypothesis when it is actually false. Increasing the sample size or the effect size can enhance statistical power, reducing the likelihood of a Type II error.

Confidence Intervals

A confidence interval (CI) provides a range of values within which the true population parameter is expected to lie. Wider confidence intervals indicate more uncertainty, while narrower intervals suggest greater precision.

Confidence intervals are closely tied to hypothesis testing: if the interval excludes the null value (e.g., zero for "no difference"), the results are considered statistically significant.

The Hypothesis-Testing Decision Framework

When conducting a hypothesis test, researchers either reject H0 or fail to reject it. Rejecting a false H0, or failing to reject a true H0, are both correct decisions. Rejecting a true H0 is a Type I error; failing to reject a false H0 is a Type II error.

Common MCAT Mistakes

• Thinking a p-value is the probability the null hypothesis is true. A p-value only measures how likely the observed data (or more extreme data) would be if H0 were true — it says nothing about the probability that H0 itself is true or false.

• Confusing Type I and Type II errors. A Type I error rejects a true H0 (false positive); a Type II error fails to reject a false H0 (false negative) — mixing these up flips which risk a study design is actually controlling.

• Assuming statistical significance means clinical or practical importance. A tiny, meaningless effect can still produce a p-value below 0.05 with a large enough sample — statistical significance only means the result is unlikely to be due to chance, not that it matters in practice.

• Believing a wide confidence interval strengthens a finding. A wider CI reflects more uncertainty about the true population parameter, not more confidence — narrower intervals indicate greater precision.

MCAT-Style Concept Check

Question: A researcher sets α = 0.05 and finds a p-value of 0.20 when testing whether a new supplement affects reaction time. What should the researcher conclude?

• A) Reject H0, because the p-value is high.

• B) Fail to reject H0, because the p-value is greater than α.

• C) Accept Ha, because the supplement definitely has no effect.

• D) The test is inconclusive and must be repeated at a lower α.

Answer: B

Explanation: The null hypothesis is rejected only when the p-value is less than the significance level. Since 0.20 > 0.05, the researcher fails to reject H0 — the data are not unusual enough under H0 to conclude a real effect exists. (A) reverses the decision rule. (C) overstates the conclusion: failing to reject H0 doesn't prove the supplement has no effect, only that this study didn't find sufficient evidence of one. (D) isn't required — a non-significant result is itself a valid, reportable outcome.

FAQ

What's the difference between the null and alternative hypothesis?

The null hypothesis (H0) states there is no effect or difference and that any observed variation is due to chance. The alternative hypothesis (Ha) proposes that a real effect or difference exists, either in a specific direction or simply that one exists at all.

What does a p-value actually tell you?

A p-value is the probability of observing the obtained results (or something more extreme) if the null hypothesis were true. A p-value below the significance level (commonly 0.05) leads to rejecting H0 as statistically significant.

What's the difference between a Type I and a Type II error?

A Type I error (false positive) rejects a true null hypothesis — concluding an effect exists when it doesn't. A Type II error (false negative) fails to reject a false null hypothesis — missing a real effect that does exist.

How does statistical power relate to Type II errors?

Statistical power is the probability of correctly rejecting a false null hypothesis. Increasing sample size or effect size increases power, which lowers the chance of committing a Type II error.