A study finds that a new teaching method produces a statistically significant improvement in test scores (p = 0.02) but the average improvement was only 0.5 points on a 100-point scale. A school board member says this is a major breakthrough. What concept is most relevant here?
- AType I error; the result may be a false positive.
- BType II error; the true effect may be much larger.
- CUndercoverage; the sample may not include all types of students.
- DStatistical significance versus practical significance; the improvement is real but too small to be educationally meaningful.Correct
Why D is right
Statistical significance (p < 0.05) indicates that the result is unlikely to occur by chance if there is no effect. However, a statistically significant result may have very small practical importance. An improvement of 0.5 points on a 100-point scale, while statistically detectable (especially in large samples), is likely too small to have educational meaning. This distinction between statistical and practical significance is crucial.
Why the other 3 choices miss
- A
- A Type I error is falsely rejecting a true null hypothesis. While possible, the most relevant concept here is not whether the result is a false positive but rather that a true small effect is not practically meaningful.
- B
- A Type II error is failing to detect a real effect (false negative). The study DID detect a significant effect, so Type II error is not the primary concern.
- C
- Undercoverage is a sampling issue. The question focuses on interpretation of a statistically significant but practically negligible result.
A researcher finds a statistically significant chi-square test for independence (p < 0.001) between gender and career preference in a sample of 10,000 adults. However, Cramer's V (a measure of effect size for categorical variables) is only 0.04. This result illustrates that
- Athe sample size of 10,000 is too large for a valid chi-square test of independence
- Bthe chi-square test must be incorrect because the reported effect size is so small
- Cwith large samples, trivial associations can still be statistically significantCorrect
- Dgender and career preference are strongly related because the p-value is below 0.001
Why C is right
With very large samples, chi-square tests have enormous power and can detect even tiny departures from independence. A Cramer's V of 0.04 indicates an extremely weak association that, while statistically significant (unlikely due to chance), has virtually no practical importance. This demonstrates the critical distinction between statistical significance and practical significance.
Why the other 3 choices miss
- A
- Incorrect. There is no upper limit on sample size for chi-square tests. The test is valid; the issue is interpreting a significant but practically meaningless result.
- B
- Incorrect. The test is not incorrect. The tiny effect truly exists in the sample; it is just too small to be meaningful. Statistical significance correctly identifies a non-zero association.
- D
- Incorrect. A very small p-value indicates that the association is unlikely to be due to chance, but it says nothing about the strength of the association. Cramer's V = 0.04 shows the association is very weak.
A medication trial finds a statistically significant reduction in LDL cholesterol (bad cholesterol) with p = 0.03. The mean reduction was 2 mg/dL. A cardiologist argues this is not clinically meaningful because typical LDL levels are 100-200 mg/dL and a 2 mg/dL reduction offers negligible benefit. Which concept is the cardiologist applying?
- AConfounding: other variables explain the LDL reduction.
- BType I error: the result is probably a false positive.
- CPractical significance: the effect, while statistically real, is too small to matter in clinical practice.Correct
- DPower analysis: the study should have been larger.
Why C is right
The cardiologist is distinguishing between statistical significance (the result is unlikely by chance, p = 0.03) and practical significance (the effect size is too small to be clinically meaningful). A 2 mg/dL reduction in LDL, while real, is far below the 10-30 mg/dL reductions typically considered clinically significant. This is the statistical vs. practical significance distinction.
Why the other 3 choices miss
- A
- Confounding involves other variables distorting the apparent effect of the treatment. The cardiologist accepts the LDL reduction is real but argues it is too small to matter.
- B
- With p = 0.03 and a true null hypothesis being unlikely to produce this result, calling it a Type I error is possible but speculative. The cardiologist's argument is about the SIZE of the effect, not whether it's a false positive.
- D
- Power analysis determines the required sample size to detect an effect of a certain size. The cardiologist is questioning whether the detected effect is meaningful, not whether the study needed more power.