The Survey’s Real Claim: Name the Population Before Judging the Numbers
When a survey conclusion feels too broad, attack the number, the respondents, or the jump from one to the other.
Rewrite the conclusion:
Most members of the target population have the reported attitude or experience.
Then inventory what the survey establishes:
- Who supplied the responses?
- What population could those respondents represent?
- What did the question measure?
- When and under what circumstances was it measured?
Compare the conclusion’s target with the survey’s actual evidence. A survey can accurately report a percentage for its respondents yet fail to support a claim about everyone else.
Do not attack a survey merely because its sample is small, recent, online, or inconvenient. Ask whether the feature changes the conclusion’s target. A weekday-only survey matters if weekdays and weekends plausibly differ. A self-selected survey matters if motivated respondents plausibly differ from nonrespondents. A question about personal experience matters if the conclusion shifts to citywide conditions.
Use this test:
- State the target population.
- State the actual respondent group.
- Identify what separates them.
- Ask whether that difference could affect the measured answer.
- Explain the likely distortion-or show why the conclusion becomes unsupported without knowing its direction.
“The respondents were unusual” is not automatically enough. The unusual feature must connect to what was measured. Answers may establish reliability about respondents without establishing representativeness of the target population.
The Crucial Distinction: Sampling Error Versus Scope Error
Having translated the survey into a population claim, classify the attack.
A sampling attack says the respondents differ from the target population in a relevant way. The conclusion generalizes from the observed group to a wider group, but selection may have favored people with particular views or experiences.
A scope attack says the survey measured one thing while the conclusion asserts another. For example:
- Respondents’ satisfaction is treated as evidence about objective quality.
- Personal experiences are treated as evidence about general conditions.
- Support for a policy is treated as evidence that the policy works.
- A current opinion is treated as evidence about a stable long-term belief.
These attacks are easy to confuse. If a survey asks, “Did the new bus route improve your commute?” then a representative sample can support a claim about residents’ reported commuting experiences. It cannot, by itself, establish that the route reduced traffic for the entire city.
Worked micro-item
A newspaper surveyed 1,000 city residents who clicked a link in an online article about a new downtown parking fee. Of those respondents, 72 percent said the fee was unfair. The newspaper concluded that most city residents consider the fee unfair.
Practice
Which one of the following, if true, most weakens the newspaper’s conclusion?
Run the Attack in Real Time: Population, Measurement, Connection
On test day, write three labels mentally:
- Target: Who does the conclusion describe?
- Measured: Who answered, and what did the question measure?
- Connection: Why might the measured result fail to apply to the target?
Inspect each answer choice for a completed connection. The strongest choice makes the survey’s number less transferable to the conclusion.
High-Yield Points - ⚡ Biggest Takeaways
Compare the survey’s target population with its actual respondents before trusting its percentage. A relevant selection difference matters only when it could affect the measured answer. Attack scope when the survey measures one property but the conclusion asserts another.
Unlock the full lesson and continue reading
Signup to continue reading this lesson and unlimited access questions, flashcards, AI notes, and more