In a hypothetical study, 4,000 adults with mild insomnia were randomly assigned to take either a melatonin supplement or a placebo before bed. Participants given melatonin fell asleep faster than those given the placebo (p = .03). A colleague points out that the effect size was d = 0.08, about one minute of difference in time to fall asleep. Which concept best explains the colleague's concern?
- AA statistically significant result can reflect an effect too small to matter in practice, especially with a large sampleCorrect
- BA p-value of .03 means there is only a 3% chance that melatonin has no real effect on sleep onset
- CThe study lacks external validity because adults' circadian rhythms differ from those of adolescents
- DBecause p < .05, the null hypothesis should be rejected, which shows that melatonin's effect on sleep onset is large and meaningful
Why A is right
This item tests the difference between statistical significance and practical significance (effect size), set in Unit 1 content on sleep. Melatonin is a hormone that helps regulate the circadian rhythm and the onset of sleep. The p-value of .03 shows that the difference in time to fall asleep is unlikely to be due to chance alone. With 4,000 participants, though, even a tiny difference can reach significance. An effect size of d = 0.08, about one minute, is far below the conventional 'small' effect of d = 0.2, so the effect may be too small to matter to people with insomnia. The colleague is pointing out that statistical significance does not guarantee practical significance.
Why the other 3 choices miss
- B
- Incorrect because a p-value is not the probability that the hypothesis is false. p = .03 means that if melatonin had no effect, a difference this large would appear by chance only about 3% of the time.
- C
- Incorrect because the colleague is not questioning whether the results generalize to other populations. The concern is about how large the effect on sleep onset is, not about who was studied.
- D
- Incorrect because rejecting the null hypothesis shows only that the difference is unlikely to be due to chance. It does not show that the effect is large or meaningful, and that gap is exactly what the colleague is pointing out.