September 29, 2000 Page One Feature A Close Contest, Plunging Responses Tax Already-Inexact Science of Polls By JOHN HARWOOD and CYNTHIA CROSSEN Staff Reporters of THE WALL STREET JOURNAL If you really want to touch off a debate about the 2000 presidential campaign, just start talking about the polls. Consider two surveys of the nation's "likely voters" that were released just this week. According to the Los Angeles Times, Texas Gov. George W. Bush has taken a solid 48%-to-42% lead over Vice President Al Gore. Then there's the poll conducted by International Communications Research, Media, Pa., which shows the same six-percentage-point advantage -- but for Mr. Gore. A Field Guide to the Campaign 2000 Polls That sort of disparity has become common as Messrs. Gore and Bush enter the final phase of their neck-and-neck race to succeed President Clinton, and it may be further magnified next week as the campaigns and the media scramble to measure the impact of next Tuesday's first debate between the two candidates. The result -- for pollsters, politicians and the ordinary voter -- has been a swelling sense of confusion, embarrassment or even outright disbelief. "Polls are often governed by the way you ask the questions," complains retired banker Frank Storey, a Republican who attended a rally for Mr. Bush this week in Spokane, Wash. Back on Capitol Hill, Senate Democratic leader Thomas Daschle sounds just as wary: "I don't buy this margin of error of three or four percentage points, spoken so authoritatively. It all has to be taken with a grain of salt." Both the Los Angeles Times and the ICR polls this week have a four-percentage-point margin of error. Serious Consequences Nonetheless, polls have very serious consequences for the campaigns that rely on them to steer their strategies and for the news organizations that use them to gauge which candidate's strategy is working. What's often poorly understood is the host of factors -- from sheer chance, to the unique nature of this year's race, to a pollster's hunches, to news deadlines -- that can make public opinion appear to swing wildly. First among them is simply the closeness of the race, which many analysts expect to rival the photo-finishes that ended the 1960 and 1968 campaigns. It is never easy to measure the sentiments of 200 million Americans by interviewing several hundred of them. But using polling to measure slight shifts in a contest between evenly matched rivals can be excruciatingly difficult. Even more daunting to pollsters is the large segment of the electorate that is apathetic, irresolute or merely undecided this year, and whose opinions can thus fluctuate, producing widely varying survey results. The science of statistics recognizes that any poll, no matter how well-conducted, is vulnerable to 1 of 7 10/10/2000 11:13 PM
misrepresenting the actual state of public opinion to some degree. In a random survey of 1,000 Americans, for instance, the "margin of error" on any individual finding might be a few percentage points in either direction. Point of Order But the everyday practice of polling, which blends science and art, can exaggerate the problem in many ways. For example, the simple order in which questions are asked can be a factor. Last week, Voter.com's "Battleground Poll," conducted by Republican pollster Ed Goeas and Democrat Celinda Lake, stood out because it showed Mr. Bush in the lead when other national surveys gave the edge to Mr. Gore. Democratic critics fingered one potential reason: The Battleground Poll asked respondents to state their presidential preference only after having asked them to evaluate President Clinton "as a person," which may have had the effect of reminding voters of the Monica Lewinsky scandal and turning respondents away from Mr. Clinton's No. 2 man, Mr. Gore. Because "there was a question in my mind" about whether the Clinton question "polluted" the presidential-preference data, the Battleground Poll reordered its questions, says Mr. Goeas. The poll has drifted toward Mr. Gore in recent days, since the change, while other surveys have moved in the opposite direction. In theory, every U.S. household should have the same chance of being included in a national opinion poll. But not every American is equally likely to be home when polling firms are operating their telephone banks, typically between the hours of 5:30 p.m. and 9:30 p.m. in each time zone. Moreover, a growing segment of the nation's population speaks English less than proficiently, or not at all. That's why some pollsters argue that the choice of days on which a survey is conducted can skew a survey's outcome. Strategists for Mr. Gore insist that polls conducted on weekends, for unknown reasons, have produced samples tilted toward Mr. Bush. Republicans argue precisely the opposite, reasoning that a more affluent GOP electorate is more likely to be out at social events on Friday nights and Saturdays. To support their argument, Republican strategists lately have homed in on one piece of evidence: a Newsweek poll of 580 "likely voters" conducted on Thursday, Sept. 14 and Friday, Sept. 15, showing Mr. Gore ahead by 14 percentage points, far more than in other surveys; that poll has a five-percentage-point margin of error. Evans Witt, an executive at Princeton Survey Research Associates, which conducts the Newsweek poll, says there is no evidence of a partisan bias in weekend polling. Constraints of time and money can introduce even more variation. To ensure that their samples are truly random, pollsters prefer to persist in trying to contact those who initially are either unable or unwilling to participate. But that can be difficult when operating under deadline pressures. As the election draws near, some news organizations have begun doing nightly "tracking" polls, and offering a rolling, three-day or four-day average of the results of relatively small groups of respondents surveyed over the previous few nights. That technique can magnify daily changes in public sentiment. Over the past 10 days or so, for example, the CNN/USA Today/Gallup tracking poll has gone from showing a double-digit Gore lead to a slight Bush advantage. Thursday, its latest tracking poll showed the two candidates dead even at 46%, with a four-percentage-point margin of error. As so-called response rates -- the percentage of those contacted who actually answer a telephone survey -- have plummeted to around 50%, deadline pressure has prompted many surveys to limit their number of "call-backs," and simply move on to other potential respondents. The problem with that practice: Those who tend to be home and answering their telephones are "quite different" from those who aren't, says Arthur H. Miller, director of the Iowa Social Science Institute in Iowa City. Women are more likely to answer than men, for instance, and older people are more likely to respond than young people. 2 of 7 10/10/2000 11:13 PM
Pollsters typically make statistical adjustments to their results to compensate for such distortions. Yet such "weighting" of polling data, which becomes ever-more complex as Election Day approaches, can introduce fresh distortions. Some pollsters "weight" according to their predictions of the relative balance of self-described Republicans and Democrats who will cast their ballots; others say such self-descriptions vary too much to be used reliably, and weight only according to fixed characteristics such as gender or race. More controversial is the effort to distinguish so-called "likely voters" from the broader pool of those registered to vote, a segment that has expanded significantly in recent years as a result of election-law changes that register many Americans when they obtain driver licenses or apply for government benefits. The Wall Street Journal/NBC News Poll, for example, has continued to highlight results from registered voters, rather than solely from those who say they are likely to vote, based on the belief that it is difficult to predict until near Election Day who is likely to show up to vote. By contrast, Utica, N.Y.-based Zogby International, which conducts the Reuters poll, surveys only "likely voters" because it believes that produces a more finely tuned result. Vulnerable to Embarrassment Yet any statistical model of voter turnout leaves a pollster vulnerable to being embarrassed by last-minute campaign developments. Because of an unexpected surge in turnout by African-American voters, for instance, many pre-election surveys in 1998 failed to forecast Democratic gains, especially in Southern states. "Mostly it's a bunch of baloney," says Mark Mellman, a leading pollster for Democratic candidates, of attempts to predict turnout. Mr. Mellman provides clients a range of potential turnout possibilities. And that's only part of the reason why poll results can vary widely. Humphrey Taylor, chairman of the Harris Poll, a unit of Harris Interactive Inc., says many unmeasurable factors can lead polls astray even more than variations in their samples. One wild card is the sheer number of polls being conducted, which tends to increase the number of inaccurate results. Most polls report a 95% "confidence level" that their findings will be accurate within the published "margin of error." But that means that in roughly 5% of cases, the errors will be greater. "It's going to happen to all of us," says Fred Steeper, a pollster for Mr. Bush. "Sampling theory tells us that." Given the range of potential difficulties, some organizations are beginning to experiment with next-generation techniques. "We're hitting the ceiling on polling," says Chris Barnes, associate director of the University of Connecticut's Roper Center for Survey Research and Analysis. "Response rates are taking a beating. They're declining fast enough now that we have to think about alternate means of data collection." Rasmussen Research, a North Carolina subsidiary of British Internet company TownPagesNet.com, has started using automated, recorded telephone surveys that respondents participate in by touching the dialpad of their telephones. While that technique could eliminate potential "interviewer effects" -- the kind of bias that can result, for example, when male interviewers question female respondents -- critics say it is vulnerable to greater rates of nonresponse or invalid responses, such as if a child were to answer. Company President Scott Rasmussen says his firm weights its data, includes questions designed to detect invalid responses, and applies a "sanity test" to ensure against error. Another solution to high telephone refusal rates is changing the polling medium altogether. Several companies, including Harris Interactive of Rochester, N.Y., and Knowledge Networks of Menlo Park, Calif., have been experimenting with Internet polls this year. The benefits of Internet polling are enticing: 3 of 7 10/10/2000 11:13 PM
They are cheap and fast; they can include pictures or video clips; and they can reach larger numbers of people simultaneously and less intrusively than the telephone. Harris Interactive's polls usually survey around 10,000 people, while most telephone polls use samples of 1,000 or fewer. Other advantages of online polling, says the Harris Poll's Mr. Taylor, are that respondents' open-ended answers tend to be richer in detail, and they are more willing to "talk" about sensitive issues online than on the telephone. The peril of online polling is that those who are computer literate and connected to the Internet tend to be younger, whiter, better educated and more affluent than the general population, a demographic that doesn't make for a very random sample. Harris Interactive has also found that online respondents tend toward middle-of-the-road responses, rather than "strongly agree" or "strongly disagree" on questions. They are also more likely to express no opinion on some political issues. Even getting a random sample among Internet users is difficult: Currently, it isn't possible for pollsters to generate random e-mail addresses as they can do with telephone numbers. Knowledge Networks tries to solve the randomness problem by recruiting a national panel of more than 100,000 people who were first selected using random-digit dialing, a method of producing random telephone numbers. Then all the households that were selected received free hardware (WebTV) and Internet access. That means Knowledge Networks' poll includes households that don't have computers and had never been on the Internet before. In January, the company, in collaboration with CBS, used this sample to measure reactions to President Clinton's State of the Union address. Although many pollsters share the vision of the Internet as the future of polling, for now voters and the political community alike have to depend on telephone surveys. And for all the fluctuations of late, recent history provides evidence that polls produce accurate signals of likely election outcomes as voters prepare to cast their ballots and their preferences begin to harden. Analyzing final presidential election polls between 1956 and 1996, the National Council on Public Polls, a trade organization, found the average poll error on each candidate was 1.9 percentage points compared with actual election outcomes. The last big polling debacle was in 1948, when all three major polls predicted a landslide by Republican Thomas E. Dewey, who in fact was narrowly defeated by Democratic President Harry Truman. Meantime, nearly everyone involved in conducting and reporting poll results eyes the fluctuating results with considerable skepticism. That often includes candidates themselves, who may spend $50,000 to conduct a survey but shudder when they glimpse the way young, low-paid telephone interviewers actually perform the work. "It can be very disconcerting when a client goes into a phone bank and watches this stuff being done," says Garry South, top strategist for California Gov. Gray Davis. "There's 24-year-olds with backward baseball caps and tattoos on the back of their necks, and you say, 'We're going to run a campaign... based on data they're collecting?' But over the long run, it works pretty well." Write to John Harwood at john.harwood@wsj.com 1 Results are latest for each poll A Field Guide to the Campaign 2000 Polls 4 of 7 10/10/2000 11:13 PM
Pollsters: Ed Goeas, Republican, and Celinda Lake, Democrat How It Works: Polling every weeknight, releasing daily "rolling" averages of past four nights; has tended to be friendlier to Bush. Insider's View: Critics think it's too GOP-friendly in its methods, but it was the first to spot recent move back in Bush's favor. Margin of error: 3.1 pct. pts. Pollsters: In-house poll -- Susan Pinkus, director How It Works: Occasional poll, screens for likely voters. Insider's View: Broke out of the pack this week by giving Bush a six-point lead, more than the consensus at the moment. Margin of error: 4 pct. pts. Pollsters: Princeton Survey Research Associates How It Works: Weekly, or near weekly; just increased its sample size from 750 registered voters to 1,000 Margin of error: 3 pct. pts. registered voter, 766 of whom were 'likely voters.' Insider's View: Al Gore's favorite reading; it's had him far better off than other polls in recent weeks. 5 of 7 10/10/2000 11:13 PM
Pollsters: In-house poll: Michael Kagay, New York Times and Kathleen Frankovic, CBS News How It Works: Poll uses a "conversion specialist" to convince reluctant respondents to cooperate. Insider's View: Early to zero in on Gore's greater "likability" as a key to his postconvention bounce. Margin of error: 3 pct. pts. Pollsters: Peter Hart, Democrat, and Bob Teeter, Republican How It Works: Highlights registered voters now, 'likely voters' as election nears; 2,000-voter survey this month is double the size of most. Insider's View: First to spot the Gore "bounce" before and immediately after the L.A. convention; results have been in the middle of the journalistic trend line for most of the year. Margin of error: 2.2 pct. pts. Pollsters: Zogby International How It Works: Republicans think its reliance on 'likely voters,' rather than registered voters, helps them. Insider's View: Rolling daily poll of likely voters begins today. Because of its reputation as being friendlier to Republicans, its big move toward Gore in late August set off alarm bells for the GOP. Margin of error: 3 pct. pts. 6 of 7 10/10/2000 11:13 PM
Pollsters: The Gallup Organization How It Works: Tracking poll has sought out 250 'likely voters' every night since Sept. 4, three-day average released daily. Insider's View: Has shown the biggest swings lately, having Gore go from 10 points up to two points down within a recent one week span. Margin of error: 4 pct. pts. Pollsters: Opinion Dynamics How It Works: Every other week, typically polls on Wednesdays and Thursdays, most recent results released last Friday, next poll Oct. 6. Insider's View: George W. Bush's favorite reading; it's tended to show Bush in better shape than in other polls. URL for this Article: http://interactive.wsj.com/archive/retrieve.cgi?id=sb970191633629706337.djm Hyperlinks in this Article: (1) mailto:john.harwood@wsj.com Margin of error: 3 pct. pts. Copyright 2000 Dow Jones & Company, Inc. All Rights Reserved. Printing, distribution, and use of this material is governed by your Subscription Agreement and copyright laws. For information about subscribing, go to http://wsj.com Close Window 7 of 7 10/10/2000 11:13 PM