Skip to content
AP Psychology · Development and Learning

Statistical significance

Statistical significance appears in 7 reviewed AP Psychology questions and sits in Development and Learning. Below are worked examples showing exactly how the exam tests it, with the answer marked and every distractor explained.

A result is statistically significant when it is unlikely to have happened by chance alone. In psychology this usually means a p-value below .05, or less than a 5 percent chance of seeing the result if there were no real effect.

Why it matters

Significance tells you whether a result is unlikely to be due to chance alone, but not how big the effect is. With a very large sample, even a tiny difference can reach significance, so researchers also report an effect size such as Cohen's d.

How the exam tests it

Expect a study of 4,000 adults that compares melatonin with a placebo and finds p = .03 but an effect size of only d = 0.08. You must explain that a significant result can still be too small to matter in practice.

Don’t confuse it with

Practical significance asks whether an effect is large enough to matter in real life. An effect of d = 0.08 can be statistically significant with 4,000 participants, but it falls well below the d = 0.2 benchmark for even a small effect.

Statistical significance in real questions

Example 1Harder
In a hypothetical study, 4,000 adults with mild insomnia were randomly assigned to take either a melatonin supplement or a placebo before bed. Participants given melatonin fell asleep faster than those given the placebo (p = .03). A colleague points out that the effect size was d = 0.08, about one minute of difference in time to fall asleep. Which concept best explains the colleague's concern?
  1. AA statistically significant result can reflect an effect too small to matter in practice, especially with a large sampleCorrect
  2. BA p-value of .03 means there is only a 3% chance that melatonin has no real effect on sleep onset
  3. CThe study lacks external validity because adults' circadian rhythms differ from those of adolescents
  4. DBecause p < .05, the null hypothesis should be rejected, which shows that melatonin's effect on sleep onset is large and meaningful

Why A is right

This item tests the difference between statistical significance and practical significance (effect size), set in Unit 1 content on sleep. Melatonin is a hormone that helps regulate the circadian rhythm and the onset of sleep. The p-value of .03 shows that the difference in time to fall asleep is unlikely to be due to chance alone. With 4,000 participants, though, even a tiny difference can reach significance. An effect size of d = 0.08, about one minute, is far below the conventional 'small' effect of d = 0.2, so the effect may be too small to matter to people with insomnia. The colleague is pointing out that statistical significance does not guarantee practical significance.
Why the other 3 choices miss
B
Incorrect because a p-value is not the probability that the hypothesis is false. p = .03 means that if melatonin had no effect, a difference this large would appear by chance only about 3% of the time.
C
Incorrect because the colleague is not questioning whether the results generalize to other populations. The concern is about how large the effect on sleep onset is, not about who was studied.
D
Incorrect because rejecting the null hypothesis shows only that the difference is unlikely to be due to chance. It does not show that the effect is large or meaningful, and that gap is exactly what the colleague is pointing out.

Studied alongside

Other concepts from Development and Learning.