Attacking a sample or a survey

Attacking a sample or a survey

Published September 28, 2026

Attacking a sample or a survey

On this page

The Survey’s Real Claim: Name the Population Before Judging the Numbers

When a survey conclusion feels too broad, attack the number, the respondents, or the jump from one to the other.

Rewrite the conclusion:

Most members of the target population have the reported attitude or experience.

Then inventory what the survey establishes:

  • Who supplied the responses?
  • What population could those respondents represent?
  • What did the question measure?
  • When and under what circumstances was it measured?

Compare the conclusion’s target with the survey’s actual evidence. A survey can accurately report a percentage for its respondents yet fail to support a claim about everyone else.

Do not attack a survey merely because its sample is small, recent, online, or inconvenient. Ask whether the feature changes the conclusion’s target. A weekday-only survey matters if weekdays and weekends plausibly differ. A self-selected survey matters if motivated respondents plausibly differ from nonrespondents. A question about personal experience matters if the conclusion shifts to citywide conditions.

Use this test:

  1. State the target population.
  2. State the actual respondent group.
  3. Identify what separates them.
  4. Ask whether that difference could affect the measured answer.
  5. Explain the likely distortion-or show why the conclusion becomes unsupported without knowing its direction.

“The respondents were unusual” is not automatically enough. The unusual feature must connect to what was measured. Answers may establish reliability about respondents without establishing representativeness of the target population.

The Crucial Distinction: Sampling Error Versus Scope Error

Having translated the survey into a population claim, classify the attack.

A sampling attack says the respondents differ from the target population in a relevant way. The conclusion generalizes from the observed group to a wider group, but selection may have favored people with particular views or experiences.

A scope attack says the survey measured one thing while the conclusion asserts another. For example:

  • Respondents’ satisfaction is treated as evidence about objective quality.
  • Personal experiences are treated as evidence about general conditions.
  • Support for a policy is treated as evidence that the policy works.
  • A current opinion is treated as evidence about a stable long-term belief.

These attacks are easy to confuse. If a survey asks, “Did the new bus route improve your commute?” then a representative sample can support a claim about residents’ reported commuting experiences. It cannot, by itself, establish that the route reduced traffic for the entire city.

Worked micro-item

A newspaper surveyed 1,000 city residents who clicked a link in an online article about a new downtown parking fee. Of those respondents, 72 percent said the fee was unfair. The newspaper concluded that most city residents consider the fee unfair.

Practice

Which one of the following, if true, most weakens the newspaper’s conclusion?

Run the Attack in Real Time: Population, Measurement, Connection

On test day, write three labels mentally:

  • Target: Who does the conclusion describe?
  • Measured: Who answered, and what did the question measure?
  • Connection: Why might the measured result fail to apply to the target?

Inspect each answer choice for a completed connection. The strongest choice makes the survey’s number less transferable to the conclusion.

High-Yield Points - ⚡ Biggest Takeaways

Compare the survey’s target population with its actual respondents before trusting its percentage. A relevant selection difference matters only when it could affect the measured answer. Attack scope when the survey measures one property but the conclusion asserts another.

Unlock the full lesson and continue reading

Signup to continue reading this lesson and unlimited access questions, flashcards, AI notes, and more

Scan to download app

Scan to download
UNLOCK FREE ACCESS

Practice Questions: Attacking a sample or a survey

  1. 1. Delivery manager: Six weeks after our company adopted a new protocol for stacking parcels in trucks, the proportion of delivered parcels generating damage claims was 35 percent lower than during the preceding six weeks. Thus, the new stacking protocol reduced damage to parcels. Which one of the following, if true, most weakens the delivery manager's argument?

    • A. The company handled slightly fewer parcels during the six weeks after adopting the protocol.
    • B. At the same time the new protocol was adopted, the company began requiring customers to submit photographic evidence before a damage claim would be processed.
    • C. Some drivers considered the former stacking protocol easier to follow than the new protocol.
    • D. Damage claims had varied moderately from week to week before the protocol was changed.
    • E. The company provided written instructions when it introduced the new stacking protocol.
    Show answer

    Answer: B. At the same time the new protocol was adopted, the company began requiring customers to submit photographic evidence before a damage claim would be processed.

    At the same time the new protocol was adopted, the company began requiring customers to submit photographic evidence before a damage claim would be processed. - The manager treats a 35 percent drop in the proportion of parcels generating damage claims as evidence that fewer parcels were damaged. - The new photographic-evidence requirement could reduce processed claims even if the rate of actual damage stayed the same, providing an alternative explanation for the drop. The company handled slightly fewer parcels during the six weeks after adopting the protocol. - The manager compares the proportion of delivered parcels generating claims, not the total number of claims. - A slightly smaller delivery volume does not explain why a smaller share of parcels generated damage claims. Some drivers considered the former stacking protocol easier to follow than the new protocol. - Driver preference does not establish that the new protocol caused more damage or failed to prevent it. - The statement gives no evidence about whether drivers followed the new protocol or how parcels fared under it. Damage claims had varied moderately from week to week before the protocol was changed. - Moderate weekly variation shows that the claim rate was not perfectly steady before the change. - It does not account for the 35 percent decline across the six weeks after the protocol was adopted. The company provided written instructions when it introduced the new stacking protocol. - Written instructions could help drivers follow the new protocol, rather than explain the decline through an unrelated change. - This fact gives no reason to think actual parcel damage stayed the same while damage claims fell.

  2. 2. Bakery owner: Three months ago, we began giving customers a discount for ordering pastries one day in advance. Since then, the proportion of pastries left unsold each evening has declined substantially. Thus, the preorder discount reduced the proportion of pastries left unsold by enabling us to forecast demand more accurately. Which one of the following, if true, most weakens the bakery owner's argument?

    • A. At the same time, the bakery stopped producing several varieties that had historically sold poorly.
    • B. Customers using the discount generally collected their orders before the bakery's busiest afternoon period.
    • C. The bakery advertised the preorder discount primarily through messages sent to its existing customers.
    • D. Some customers who preordered pastries also purchased beverages when collecting their orders.
    • E. Competing bakeries nearby did not offer discounts to customers who ordered pastries in advance.
    Show answer

    Answer: A. At the same time, the bakery stopped producing several varieties that had historically sold poorly.

    At the same time, the bakery stopped producing several varieties that had historically sold poorly. - Discontinuing pastries that often went unsold could itself reduce the proportion left unsold each evening. - Because this change occurred when the discount began, the decline does not establish that preorders improved demand forecasts. Customers using the discount generally collected their orders before the bakery's busiest afternoon period. - Collection time does not explain why a smaller proportion of pastries remained unsold. - Picking up an order before the busiest period does not show that forecasting failed to improve. The bakery advertised the preorder discount primarily through messages sent to its existing customers. - Advertising to existing customers identifies who heard about the discount, not why fewer pastries went unsold. - Those customers’ advance orders could still have helped the bakery forecast demand. Some customers who preordered pastries also purchased beverages when collecting their orders. - Beverage purchases do not account for the decline in unsold pastries. - They also provide no reason to doubt that advance pastry orders improved forecasts. Competing bakeries nearby did not offer discounts to customers who ordered pastries in advance. - Competitors’ discount policies do not explain the bakery’s lower proportion of unsold pastries. - This fact does not challenge the owner’s proposed link between preorders and more accurate forecasts.

  3. 3. Product researcher: Participants received identical snack portions in either newly designed containers or the company's older containers. Those receiving the new containers rated the snacks as fresher. Thus, the new container design caused participants to perceive the snacks as fresher. Which one of the following, if true, most weakens the researcher's argument?

    • A. Some participants assigned to the new containers preferred the appearance of the older containers.
    • B. The new containers were stored on a warmer shelf than the older containers during the study.
    • C. Several participants in each group had previously purchased the product used in the study.
    • D. Participants were not told how much either type of container cost to manufacture.
    • E. The new containers can hold slightly more product than the older containers can hold.
    Show answer

    Answer: B. The new containers were stored on a warmer shelf than the older containers during the study.

    The new containers were stored on a warmer shelf than the older containers during the study. - The groups differed in storage temperature as well as container design, so the freshness ratings do not isolate the effect of design. - If warmer storage affected how fresh the snacks seemed, it could explain the higher ratings even though the portions were identical. Some participants assigned to the new containers preferred the appearance of the older containers. - A preference for the older containers’ appearance does not explain why the new-container group rated its snacks as fresher. - The claim concerns perceived freshness, not whether every participant liked the new design. Several participants in each group had previously purchased the product used in the study. - Prior purchases occurred in both groups, so this fact identifies no difference that explains their different ratings. - Nothing states that previous purchasers were distributed in a way that favored the new-container group. Participants were not told how much either type of container cost to manufacture. - Manufacturing cost has no stated connection to how fresh the snacks seemed. - Because neither group was told the costs, that information cannot explain the difference in ratings. The new containers can hold slightly more product than the older containers can hold. - Greater capacity does not mean participants received more snack: the portions were identical. - The capacity difference therefore provides no reason, on these facts, for the new-container group’s higher freshness ratings.

  4. 4. Product manager: After our company rewrote the planting instructions included with its seed packets, customer complaints about unsuccessful germination declined sharply. Therefore, the decline occurred because the revised instructions were clearer and helped customers avoid planting errors. Which one of the following, if true, most strengthens the product manager's argument?

    • A. Sales of the seed packets increased after the revised instructions were introduced.
    • B. Complaints about the durability of the seed packets also declined during the same period.
    • C. Several experienced gardeners praised the illustrations included with the revised instructions.
    • D. Gardeners randomly given the revised instructions made fewer planting errors than those given the former instructions.
    • E. The company's customer helpline was open for fewer hours after the instructions were revised.
    Show answer

    Answer: D. Gardeners randomly given the revised instructions made fewer planting errors than those given the former instructions.

    Gardeners randomly given the revised instructions made fewer planting errors than those given the former instructions. - Random assignment makes the groups comparable, so the difference in planting errors supports the claim that the revised instructions caused it. - Fewer errors directly support the manager’s proposed explanation for the decline in complaints about unsuccessful germination. Sales of the seed packets increased after the revised instructions were introduced. - Higher sales do not show that customers understood the revised instructions better or made fewer planting errors. - The argument concerns why complaints declined, not how many packets were sold. Complaints about the durability of the seed packets also declined during the same period. - Packet durability is separate from planting errors and unsuccessful germination. - A decline in another kind of complaint does not establish that clearer instructions caused the decline in germination complaints. Several experienced gardeners praised the illustrations included with the revised instructions. - Praise from experienced gardeners does not show that typical customers found the planting directions clearer. - It also does not show that the illustrations reduced planting errors. The company's customer helpline was open for fewer hours after the instructions were revised. - Fewer helpline hours could mean customers had fewer opportunities to complain, even if germination did not improve. - That alternative explanation weakens the inference that clearer instructions caused the decline in complaints.

  5. 5. A cinema sold more tickets this month than in any previous month. Yet the percentage of available seats occupied at its showings was lower this month than during any previous month. Which one of the following, if true, most helps to resolve the apparent discrepancy?

    • A. The cinema opened several additional auditoriums, increasing the number of seats available at each showing.
    • B. The cinema sold fewer tickets for weekday showings but more tickets for weekend showings.
    • C. Some customers bought tickets online rather than at the cinema's ticket counter.
    • D. The cinema showed more films that had already been screened at other local cinemas.
    • E. Customers who attended the cinema purchased fewer refreshments per visit than before.
    Show answer

    Answer: A. The cinema opened several additional auditoriums, increasing the number of seats available at each showing.

    The cinema opened several additional auditoriums, increasing the number of seats available at each showing. - The occupancy percentage compares tickets sold with seats available, so record ticket sales can coincide with a record-low percentage if capacity rises even faster. - Additional auditoriums supply the missing change in available seats that reconciles the two facts. The cinema sold fewer tickets for weekday showings but more tickets for weekend showings. - This shifts sales between weekdays and weekends but does not establish an increase in available seats. - Without such an increase, the shift does not explain how record total sales produced the lowest overall occupancy percentage. Some customers bought tickets online rather than at the cinema's ticket counter. - Buying online changes the purchase method, not the number of tickets sold. - It also leaves the number of available seats unchanged, so it cannot explain the lower occupancy percentage. The cinema showed more films that had already been screened at other local cinemas. - Where the films were previously screened does not establish a change in the cinema’s seating capacity. - It therefore does not reconcile record ticket sales with a record-low occupancy percentage. Customers who attended the cinema purchased fewer refreshments per visit than before. - Refreshment purchases do not enter the occupancy percentage, which concerns tickets and available seats. - Buying fewer refreshments therefore explains neither the record ticket sales nor their coexistence with lower occupancy.

  6. 6. Public-records director: After our office introduced an online request portal, the total number of requests completed each month increased, and the number of requests pending for more than 30 days decreased. Yet the average employee time spent on each request handled by employees increased. Which one of the following, if true, most helps to resolve the apparent discrepancy?

    • A. A new portal automatically completed routine requests, freeing employees to clear old cases while leaving them unusually complex new cases.
    • B. Employees began recording preparation time that had previously been excluded from the calculation of time spent per request.
    • C. Most people submitting requests through the new portal had never submitted a public-records request before.
    • D. Several employees who processed requests also began answering questions from people considering submitting requests.
    • E. The office received more requests concerning recent records than concerning records created many years earlier.
    Show answer

    Answer: A. A new portal automatically completed routine requests, freeing employees to clear old cases while leaving them unusually complex new cases.

    A new portal automatically completed routine requests, freeing employees to clear old cases while leaving them unusually complex new cases. - Automatic completion increased total monthly completions and freed employees to reduce the number of requests pending more than 30 days. - Because employees handled a greater share of complex requests, their average time per employee-handled request could rise even as the office completed more requests overall. Employees began recording preparation time that had previously been excluded from the calculation of time spent per request. - Counting previously excluded preparation time could raise the reported average without any change in how long requests actually took. - It does not explain why monthly completions increased or why fewer requests remained pending more than 30 days. Most people submitting requests through the new portal had never submitted a public-records request before. - The requesters’ prior experience does not establish how much employee time their requests required. - It also does not explain the increase in completions or the decrease in requests pending more than 30 days. Several employees who processed requests also began answering questions from people considering submitting requests. - Answering prospective requesters’ questions adds a duty but does not show that employees spent more time on each request they handled. - The added duty does not explain how the office completed more requests and reduced its older backlog. The office received more requests concerning recent records than concerning records created many years earlier. - Record age alone does not show whether a request is routine, complex, or time-consuming for employees. - This comparison does not account for both the improved completion and backlog figures and the higher average employee time per handled request.

  7. 7. Research director: Our institute began offering monthly public webinars summarizing its working papers. During the following six months, downloads of those papers increased substantially. Thus, offering the webinars caused the increase in downloads. Which one of the following, if true, most weakens the research director's argument?

    • A. Most webinar attendees watched only portions of the presentations.
    • B. Several working papers discussed subjects not covered in the webinars.
    • C. A major academic search engine began indexing the institute's working papers during the same period.
    • D. Researchers who downloaded the papers could access them without attending a webinar.
    • E. Attendance at the webinars declined somewhat during the six-month period.
    Show answer

    Answer: C. A major academic search engine began indexing the institute's working papers during the same period.

    A major academic search engine began indexing the institute's working papers during the same period. - Search-engine indexing could have made the papers easier to find, providing an alternative cause of the increased downloads. - Because indexing began during the same six months as the webinars, the timing alone does not establish that the webinars caused the increase. Most webinar attendees watched only portions of the presentations. - Even a partial presentation could prompt an attendee to download a working paper. - This does not explain the increase in downloads independently of the webinars. Several working papers discussed subjects not covered in the webinars. - The webinars could still have increased downloads of the papers they covered. - The director claims an increase in downloads overall, not that every paper received a webinar. Researchers who downloaded the papers could access them without attending a webinar. - Access without attendance does not show that the webinars failed to draw attention to the papers. - Someone could learn about a paper through a webinar and download it afterward. Attendance at the webinars declined somewhat during the six-month period. - A somewhat declining attendance trend does not rule out the webinars as a source of new readers. - Unlike search-engine indexing, this fact supplies no separate cause for the substantial rise in downloads.

  8. 8. A ferry operator's records show that the total number of passenger trips taken this year was 8 percent lower than last year, yet total passenger-fare revenue was 6 percent higher. The ferry's schedule of ticket prices did not change between the two years. Which one of the following, if true, most helps to resolve the apparent discrepancy?

    • A. A larger proportion of this year's passengers bought full-price single-trip tickets instead of discounted multi-trip passes.
    • B. The ferry operator spent less on advertising this year than it spent during the preceding year.
    • C. Several passengers who used the ferry last year moved away before the beginning of this year.
    • D. The ferry operator added newer vessels while retiring an equal number of older vessels.
    • E. Some passengers consider the ferry more comfortable than other available forms of transportation.
    Show answer

    Answer: A. A larger proportion of this year's passengers bought full-price single-trip tickets instead of discounted multi-trip passes.

    A larger proportion of this year's passengers bought full-price single-trip tickets instead of discounted multi-trip passes. - The unchanged price schedule does not mean the average fare per trip stayed the same: a shift toward full-price tickets could raise it. - That higher average fare could make total fare revenue rise 6 percent even as passenger trips fell 8 percent. The ferry operator spent less on advertising this year than it spent during the preceding year. - Lower advertising spending concerns an expense, not the passenger-fare revenue in the records. - It does not explain how fewer passenger trips generated more fare revenue. Several passengers who used the ferry last year moved away before the beginning of this year. - Departing passengers could help explain the 8 percent decline in trips. - Their departure does not explain why fare revenue rose despite that decline. The ferry operator added newer vessels while retiring an equal number of older vessels. - Replacing vessels says nothing about the fares collected per trip. - Without a link to ticket purchases, the fleet change does not reconcile fewer trips with higher fare revenue. Some passengers consider the ferry more comfortable than other available forms of transportation. - A preference for comfort does not identify a change in the mix of tickets purchased. - It therefore does not explain why fare revenue rose while the number of trips fell.

Flashcards: Attacking a sample or a survey

  1. 1. A study uses a small, self-selected group to support a conclusion about an entire population. What fact would most directly weaken the conclusion?

    Show answer

    The participants differ from the overall population in a way relevant to the measured result. That makes the sample unrepresentative.

  2. 2. A survey concludes that most residents support a policy. How could the wording of the survey weaken that conclusion?

    Show answer

    Show that the question was leading, ambiguous, or omitted an important cost of the policy. The responses may then reflect the wording rather than genuine support.

  3. 3. When should you attack the 'scope' of a survey argument?

    Show answer

    Attack the scope when the survey measures one specific property or condition, but the conclusion of the argument asserts a claim about a different, broader, or unrelated property or condition.

  4. 4. In the street trees and crime rate example, what role does 'wealth' play in weakening the conclusion?

    Show answer

    Wealth acts as an alternate cause. It explains both the presence of street trees (wealthier neighborhoods can afford them) and lower crime rates (wealthier neighborhoods spend more on security), thus making the conclusion that trees reduce crime less necessary.

  5. 5. What are the three biggest takeaways for Weaken-EXCEPT questions?

    Show answer

    1. Mark four genuine attacks and choose the single answer that leaves the exact conclusion intact. 2. Judge every choice against the conclusion’s precise scope. 3. A consequence consistent with the conclusion is not weakening.

  6. 6. Before evaluating a potential weaken answer, what is the crucial first step to prevent misidentifying the target?

    Show answer

    Mentally or physically label the exact conclusion (e.g., 'Causal claim: X caused Y,' 'Recommendation: We should do X') to ensure the answer attacks that specific claim and not an unstated one.

  7. 7. An argument infers that a new procedure caused improvement because results were better after its adoption. What comparison would most strongly weaken that inference?

    Show answer

    Show that a similar group that did not adopt the procedure improved by the same amount. This points to a broader factor rather than the procedure as the cause.

  8. 8. When weakening an argument, should you dispute a stated premise or attack the reasoning from the premises to the conclusion?

    Show answer

    Accept the premises as given and attack the link to the conclusion. Show that the premises can be true while the conclusion is less likely to follow.

Everything you need for LSAT prep

Get full Oncourse access with lessons, practice questions, flashcards and AI study tools.

GET STARTED FOR FREE
75% OFFLimited time offer
GET 75% OFF