A study finds that a new teaching method produces a statistically significant improvement in test scores (p = 0.02) but the average improvement was only 0.5 points on a 100-point scale. A school board member says this is a major breakthrough. What concept is most relevant here?
- AType I error; the result may be a false positive.
- BType II error; the true effect may be much larger.
- CUndercoverage; the sample may not include all types of students.
- DStatistical significance versus practical significance; the improvement is real but too small to be educationally meaningful.Correct
Why D is right
Statistical significance (p < 0.05) indicates that the result is unlikely to occur by chance if there is no effect. However, a statistically significant result may have very small practical importance. An improvement of 0.5 points on a 100-point scale, while statistically detectable (especially in large samples), is likely too small to have educational meaning. This distinction between statistical and practical significance is crucial.
Why the other 3 choices miss
- A
- A Type I error is falsely rejecting a true null hypothesis. While possible, the most relevant concept here is not whether the result is a false positive but rather that a true small effect is not practically meaningful.
- B
- A Type II error is failing to detect a real effect (false negative). The study DID detect a significant effect, so Type II error is not the primary concern.
- C
- Undercoverage is a sampling issue. The question focuses on interpretation of a statistically significant but practically negligible result.
In a hypothesis test, a researcher obtains a p-value of 0.03. She reports: "There is a 3% chance that the null hypothesis is true." What is wrong with this interpretation?
- AThe p-value should be compared to alpha, not interpreted as a probability
- BA p-value of 0.03 is not statistically significant
- CNothing; this is a correct interpretation of the p-value
- DThe p-value is the probability of observing data as extreme as (or more extreme than) the sample data, given that the null hypothesis is true — not the probability that H0 is trueCorrect
Why D is right
The p-value is the probability of obtaining a test statistic as extreme as the observed value, assuming H0 is true. It is NOT the probability that H0 is true. The null hypothesis is either true or false; the p-value quantifies the strength of evidence against H0, not the probability of H0 being true.
Why the other 3 choices miss
- A
- Incorrect because the p-value is indeed a probability and can be interpreted, but it must be interpreted correctly as described in the correct answer.
- B
- Incorrect because 0.03 < 0.05, making it statistically significant at the conventional alpha = 0.05 level.
- C
- Incorrect because this is one of the most common misinterpretations of p-values.
A student carries out a chi-square test of independence on a two-way table that classifies 150 randomly selected students by grade level (3 categories) and by whether they prefer a printed or digital textbook (2 categories). The conditions for inference are met. The test gives chi-square = 4.2 with 2 degrees of freedom and a p-value of about 0.122.
Suppose instead that 300 students had been selected and every observed cell count was exactly double the original count, so all the sample proportions are unchanged. How would the results of the test change?
- AThe statistic stays 4.2 because the cell proportions are unchanged, so the p-value stays about 0.122.
- BThe statistic doubles to 8.4, but the degrees of freedom double to 4, so the p-value stays about 0.122.
- CThe statistic doubles to 8.4 and the p-value decreases.Correct
- DThe statistic halves to 2.1 because the expected counts double, so the p-value rises above 0.122.
Why C is right
When every observed count is doubled, every expected count also doubles. Each term (O - E)^2 / E becomes (2(O - E))^2 / (2E) = 2(O - E)^2 / E, so the chi-square statistic doubles from 4.2 to 8.4. The degrees of freedom depend only on the table's dimensions, (2 - 1)(3 - 1) = 2, so they do not change. With df = 2, the p-value for 8.4 is about 0.015, which is smaller than 0.122: the same proportions give stronger evidence of an association when they come from a larger sample.
Why the other 3 choices miss
- A
- Assumes the chi-square statistic depends only on proportions. It actually depends on counts, so the same proportions from a larger sample give a larger statistic.
- B
- Thinks degrees of freedom depend on sample size. For a two-way table, df = (rows - 1)(columns - 1) = 2 no matter how large n is.
- D
- Divides by the larger expected counts but forgets that the squared differences (O - E)^2 grow by a factor of 4. The net effect is that the statistic doubles, not halves.