Survey Research (Polling)

Survey Research (Polling)

Types of Surveys personal (face-to-face) interviews: high response rate, can ask more questions, but... expensive, takes longer to administer telephone interviews: cheaper, fairly easy to get representative sample (especially RDD, some lists), but... cell phones, class bias, respondent fatigue mail surveys: much cheaper, access to elite or other specialized populations, but... slow, unclear who is filling out questionnaire internet surveys: opportunity for more creativity (experimental designs, visual and interactive elements), but... some loss of representativeness (class? age?), unclear who is doing the survey

Whatever method is used, survey research is equal parts science (drawing a sample that is representative of the larger population of interest) and art (designing the questions to be asked). The goal is to reduce (not to eliminate, which is impossible) two things: sampling error (the science) measurement error (the art)

sampling error: the discrepancy between an observed value (in your sample) and a true value (in the overall population) that arises purely from chance Cannot be eliminated because no sample is a perfect representation of the larger population from which it was drawn. Question: How much sampling error are we willing to tolerate? A little (campaign pollsters whose clients want fairly precise numbers), slightly more (researchers who study broad trends in attitudes and behavior), or maybe a lot (media polls, where accuracy isn t as important as getting a good story)?

Sampling Horror Stories (election version) Elections are the only verifiable opinion polls: The Literary Digest (1936) Gallup and others (1948, compare with 1980) challenge of identifying likely voters (polling house differences) e.g., Gallup mid-course adjustments in 1992 and 2000

David Moore, The Opinion Makers (2008), p. 77. Note the wide swings from one poll to the next, especially in September. Not clear whether all polls are of likely voters if not, that could be one explanation for the differences.

The amount of sampling error (which can be calculated fairly precisely for probability samples) basically depends on three factors: how respondents are selected (the closer to a simple random sample, the better) sample size (bigger is better, but with diminishing returns; sampling method is more important, see The Literary Digest 1936) sampling fraction, or the proportion of the total population that is included in the sample (not a major factor with large populations because the fraction is always low)

Probability vs. Nonprobability Samples Examples of nonprobability samples: standing on the corner of University Ave. and 13 th Street, interviewing every person (or every n th person) who walks by internet or radio/tv surveys where respondents are selfselected constituent mail questionnaires conducted by members of Congress and other elected officials pure quota sampling (as in 1948), where the social/demographic makeup of the sample is matched to that of the target population but interviewers can select for themselves the actual people with whom they will speak

A probability sample is where everyone in your target population has a nonzero chance of being selected and the pollster can determine the actual probability of a person being selected (which allows the use of statistical formulas to estimate how likely it is that the attitudes of those in the sample will match the attitudes of the target population as a whole). In a simple random sample (roughly comparable to drawing names out of a hat), everyone in the population has an equal chance of being selected. These are uncommon even with today s phone and internet surveys because they require a listing of the entire target population something that is not always available (e.g., likely voters ). Result: Somewhat more sampling error than with a pure random sample.

Applying probability theory to survey research, two ideas are especially important: margin of error: the range of values within which true population values are likely to fall confidence level: the degree of belief that an estimated range of values (above) includes the true population value To illustrate...

X number of people flip a coin 1,000 times each (with an equal probability for heads or tails on any single flip, as opposed to using a loaded coin). The result will probably be a normal-curve distribution with some outliers (those who generate a disproportionate number of heads or tails), but most observations near (not necessarily right at) 500 heads and 500 tails.

This represents a distribution of samples (each person flipping represents/generates a different sample): As the number of samples increases, the distribution (of coin flips) for those samples should begin to approximate the distribution of the total population (50-50, at least in principle). Applied to opinion surveys, for example, if the true number of self-identified liberals among voting-age adults is 25%, then... the more surveys of voting-age adults that one conducts, the closer the mean estimate of liberals based on those samples should be to 25%.

This means that many (most) samples will have estimates that deviate from 25% by a little or even a lot not because the samples weren t drawn randomly, but because (by chance) many of them will contain larger numbers of liberals and others fewer than are present in the population. Such error cannot be eliminated altogether except by interviewing the entire population (which is seldom possible). For academic research, an acceptable confidence level (the degree of belief that an estimated range of values includes the true population value) is usually 90 or 95 percent. Range of values? What s that?

This takes us to confidence level, or the degree of belief (confidence) that an estimated range of values includes the true population value. Example: If 50 percent of our survey respondents express approval of the president s job performance; and the survey s margin of error is ±3 percentage points, then... it is likely that the true population value (which we would observe if the entire population were surveyed) lies between 47 and 53 percent approval.

How likely is it? That depends on the confidence level (which tells us the probability that our sample is un/representative of the larger population from which it is drawn). And what determines the range of values (margin of error)? Basically the same things that determine confidence level: sampling method (random is better) and sample size (more respondents, less error). Putting everything together: For a simple random sample, at the 95 percent confidence level...

±0.05 ±0.03 ±0.01 Population size 1,000 278 516 906 20,000 377 1,013 6,488 500,000 384 1,065 9,423 2,000,000 384 1,067 9,558 Translation: If the population size is 1,000, and if you draw 100 samples from that population, in 95 of those 100 samples the true population value should fall within ±5 points of the observed value for the sample if you interview 278 of the 1,000 individuals who comprise the larger group. If you want greater accuracy, you ll need to interview more people (516 of 1,000 for ±3 points, 906 of 1,000 for ±1 point). Note: 5 of the 100 samples will have observed values that fall outside this range. And so on...

The questions that must be answered by the researcher are therefore: How much error are we willing to tolerate? How much accuracy can we afford? Be aware that margin of error increases, often quite dramatically, for subgroups whose attitudes you may want to know about (based on race, gender, income, and so on); and questions on which large numbers of respondents express no opinion (thereby effectively reducing your sample size for that question).

Consumers of surveys should also take into account the existence of measurement error, which results from the fact that no single survey question (or series of questions), however wellconstructed, can perfectly capture a person s underlying attitude or belief. At best, the answers people give to our questions are approximations of the true attitude that we re trying to measure.

Some sources of measurement error: interviewer effects (including accidental misrecording of answers) respondent fatigue (in longer interviews) response set (repeatedly giving the same answer, such as agree or disagree, to a series of forced-choice questions) social desirability (failing to report socially unacceptable behavior, e.g., racial bigotry, nonvoting)

nonattitudes (meaningless answers given by someone with no true opinion on a subject) question order/context (answers artificially influenced by the sequence in which questions are asked) in general, poorly worded questions (e.g., that are vague, biased, or confusing to the respondent) One strategy for reducing measurement error (response set, nonattitudes) is to ask open-ended questions. But... these are time-consuming (meaning the survey either contains fewer questions or is more expensive), difficult to analyze (increasing the risk of interpretive error), and more fully capture the attitudes of articulate and bettereducated respondents.

Example of a bad question (from a 1994 survey sponsored by the American Foundation for AIDS Research): The AIDS epidemic is a national emergency. It has already claimed over 180,000 lives in the U. S. alone. Over one and a half million Americans now carry the AIDS virus. Do you think the majority of Americans realize how widespread this tragedy has become, and that the worst is still ahead? Problems: argumentative provides information that many people don t have, thereby making respondents unrepresentative of the population from which they were drawn (Moore) compound question

Another very bad question (data from two 1992 Roper polls conducted for the American Jewish Committee): Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened? (22 percent said possible, plus 12 percent weren t sure) Problem: double negative Better: Does it seem possible to you that the Nazi extermination of the Jews never happened, or do you feel certain that it happened? (only 1 percent said possible)

A more subtle example (from two July 1992 polls on candidate favorability): Gallup asked people whether their views of Bill Clinton were favorable (63 percent) or unfavorable (25 percent), with 12 percent volunteering that they had no opinion. CBS/NYT gave respondents four options to choose from: favorable (36 percent), unfavorable (24 percent), undecided (31 percent), and haven t heard enough about the candidate to offer an opinion (9 percent). Moral: Changes in question wording and response format can make a big difference.

From surveys done back in the 1940s: Two questions asked: Do you think the United States should let Communist newspaper reporters from other countries come in here and send back to their papers the news as they see it? Do you think a Communist country like Russia should let American newspaper reporters come in and send back to America the news as they see it? Support for Q1 was much lower when it was asked first. Moral: Question order (and context) also can make a big difference.

Gallup split-half survey (2010): Question A: Are you in favor of the death penalty for a person convicted of murder? Yes: 64% No: 29% No opinion: 6% Question B: If you could choose between the following two approaches, which do you think is the better penalty for murder the death penalty or life imprisonment with absolutely no possibility of parole? Death penalty: 49% Life in prison: 46% No opinion: 6%

Framing effects in surveys (circa 2006; Clawson-Oxley p. 34): Baseline: Do you strongly oppose, somewhat oppose, somewhat favor, or strongly favor the death penalty for persons convicted of murder? 65% of white respondents either strongly or somewhat favored vs. 50% of blacks Racial frame: Some people say that the death penalty is unfair because most of the people who are executed are African Americans. Do you strongly oppose...? whites 77% vs. blacks 38% Innocence frame: Some people say the death penalty is unfair because too many innocent people are being executed. Do you strongly oppose...? 64% whites vs. 34% blacks

Support for divided government: Do you think it is better when one party controls both the presidency and Congress, better when control is split between the Democrats and Republicans, or doesn't it matter? Some people think it is better when one party controls both the White House and Congress, while others feel that it's better when control is split between the Republicans and the Democrats. What about you... When the president is a Republican, do you prefer that the Democrats control Congress or the Republicans control Congress? [Repeat for president a Democrat.]

2000 ANES: divided government unified party control doesn t matter not sure 51 percent 23 percent 23 percent 2 percent 2001 UF/PC national survey: divided government 33 percent unified party control 15 percent Republican Congress regardless 17 percent Democratic Congress regardless 16 percent doesn t matter 7 percent other combination 12 percent

Is there anything wrong with these questions? Why did you vote for Obama/Romney in the last election? Why do you consider yourself to be a liberal/conservative [or Democrat/Republican]? Why did you decide not to vote in the last election? Answer: Yes!! Because people often are unaware of the reasons why they think, feel, or behave the way they do and the answers they give when you ask may be misleading even if they don t intend them to be.

Nonattitudes David W. Moore, The Opinion Makers (Beacon Press, 2008): Standard: Would you favor or oppose sending American ground troops to the Persian Gulf in an attempt to remove Saddam Hussein from power in Iraq? Follow-up: Depending on their answer, respondents were asked whether they would be upset if government did/did not send troops to Iraq; those who said no were coded as DK/unsure.

More of the same: For vote preference questions, respondents were initially asked whether they had definitely decided which candidate to support; those who said they were still considering the candidates are counted as undecided in version 2.

Push Poll: a survey-type instrument containing questions, the goal of which is not to gain information but rather to change the opinion of likely voters, who are contacted in larger numbers than is necessary for a valid campaign survey. This is accomplished by divulging negative (often false) information about the opponent that is designed to push the voter away from him/her and pull the voter toward the candidate who is paying for the calls.

Examples: Maine (1994) Respondents were asked whether their opinion of a U. S. House candidate would change if they knew he had defaulted on $10k worth of student loans, yet had loaned the same amount to his campaign. In fact, the candidate still owed $7k on his student loans but had always met his required monthly payment and otherwise abided by all terms of the loan agreement.

Democratic primary for governor of Ohio (1982): As you may know, in 1974, [former mayor of Cincinnati] Jerry Springer, who had gotten married six months earlier, was arrested on a morals charge with three women in a hotel room. He also used a bad check to pay for the women s services, and subsequently resigned as mayor of his city. Does this make you much more likely, somewhat more likely, somewhat less likely, or much less likely to support Jerry Springer for governor this year?

Push polls are not polls at all, the sole purpose being to sell a product (candidate) to a targeted group of voters; usually done in the closing weeks/days of a campaign, when they are less likely to be detected (either by the opposing candidate or the press) and there is less time for an effective response; disguised, with the identity of caller (polling firm, party, candidate, independent group) not revealed and therefore nearly impossible to trace (unless somebody is able to record the conversation); typically based on false information about the target candidate.

Assignment: Find one poll (or story about a poll) online that is problematic in terms of either sampling or measurement. Discuss in class.