Sampling and Non Response Biases in Election Surveys : The Case of the 1998 Quebec Election

Transcription

1 Sampling and Non Response Biases in Election Surveys : The Case of the 1998 Quebec Election Presented at the International Conference on Survey Non response, held in Portland, Oregon, October By Claire Durand, dept. of sociology, U of Montreal, Andre Blais, dept. of political science, U of Montreal, Sebastien Vachon, dept. of sociology, U of Montreal Contact First author at: Claire Durand durandc@socio.umontreal.ca Dept. of sociology, University of Montreal, C.P. 6128, succ. Centre-ville, Montreal, Quebec, H3C 3J7 Note: We wish to thank the survey companies, and most specially Crop and Createc, for their cooperation to this research. We also wish to thank the FCAR and SSHRC for financial support.

2 Abstract : During the last electoral campaign in Quebec, Canada, all the polls published in the media had a similar estimate of vote intentions. The Parti Quebecois (PQ), a centre-left party dedicated to Quebec sovereignty, was clearly ahead, by an average of five points in the last six polls of the campaign. The PQ won the election, held on November 30, 1998, but with a lesser share of the vote (43%) than the contending Liberal Party (44%), a centre-right federalist party. Pollsters and many observers have contended that the discrepancy between the polls and the actual vote could be explained either by a last minute shift in favor of theliberals or by differential turnout. We rely on a number of sources of data in order to sort out the possible causes for such a discrepancy. A post election poll was conducted among fifteen hundred respondents who had answered one of three electoral surveys conducted during the penultimate week of the campaign by two Quebec pollsters (CROP and CREATEC). The response rates for the campaign surveys varied from 50% to 60% and the reinterview rate in the post election survey was 83%. An analysis of the data from three surveys carried out by Crop during the fourweeks campaign was performed in order to estimate the impact of item and survey non response. A study of voting sections with a high percentage of collective households allows us to estimate the voting behavior of residents of collective households. Two Statmedia studies conducted in 1997 and 1998 provided information on the sociodemographic characteristics of respondents from unlisted and doubly listed telephone lines. Finally, three Crop surveys carried out after the election allows us to compare the voting intentions of respondents from listed and unlisted telephone numbers. The results of the post election survey do not support the late shift and differential turnout hypotheses. The most likely explanation for the discrepancy between vote intentions as revealed in the polls and the actual vote is to be found in sampling and non response biases. Analysis of item non response as well as survey non response shows that there is a consistent tendency for non respondents to be supporters of the Liberal Party. An analysis of sampling frame biases also shows that Liberal supporters are likely to be under sampled. Finally, adjustment weighting also tends to increase the bias against the Liberal Party. It is pointed out that these biases are not specific to the Quebec situation and are likely to increase with demographic and technological changes.

3 Contents 1.0 Context of the study Methodology The first hypothesis : the electorate moved Was it non response? Item non response Survey non response Hard to reach households and individuals Non cooperative households or individuals Non respondents Sampling frame issues Unlisted telephone numbers Doubly listed phone numbers Collective households Sampling frames and weighting Conclusion... 17

4 The general purpose of this article is to trace the possible effects of non response on the estimation of vote intentions. The case of the election held in November 1998 in Quebec, Canada, is examined. During the 30 day electoral campaign, 17 polls were published by the media. Only three of these put the Quebec Liberal Party, a center-right federalist party, ahead of the Parti Québécois, a center-left party dedicated to Quebec sovereignty. The last six polls of the campaign gave, on average, a 5-point lead to the Parti Québécois over the Liberal Party. On election day, it turned out, however, that the Parti Québécois had been outvoted by the Liberal Party (44 p. cent of the vote vs. 43 p. cent). The first reaction of most pollsters and academics was to attribute this situation to the electorate (late campaign swing, differential turnover). Other academics argued that the gap between the polls and the actual outcome of the election could be due to non response and/or to sampling frame biases. The paper assesses the plausibility of these various explanations. 1.0 Context of the study Three main areas of research have developed in order to explain discrepancies between the measures of voting intentions by the polls and the outcomes of the vote. The first area of research is related to the electorate : either it changed its mind between the time when the survey was conducted and the vote or turnout is not proportionally distributed among party supporters. This area of research has driven pollsters to conduct surveys till the end of electoral campaigns, in order to explore the possibility of late campaign shifts. 1

5 Hypotheses attributing discrepancies to late campaign changes in the electorate have been examined by a number of authors. The 1992 election in Great Britain is one modern case that has drawn attention. Jowell et al. (1993) have shown that moves in the electorate and differential turnout, together with item non response, could at best explain half of the nine points discrepancy between the polls and the vote. A number of authors have examined similar situations where discrepancy appeared between polls estimates of vote intention and the outcomes of the vote (Howell and Simms, 1994; Curtice 1997, Bishoping and Schuman, 1994; Traugott and Price, 1992). The discrepancy seems to always head in the same direction, that of an under representation of the more conservative vote in the polls. Validation of substantial late moves that could explain discrepancies have yet to be found though shifts between parties do indeed occur (Jowell et coll., 1993). When the examination of all the available surveys of a campaign shows no move from one week to the other, whether using traditional mean difference or time series analysis, it is unlikely that a shift occurred (Erikson and Wlezien, 1999) in the last few days unless an important event could explain such a shift. Differential turnout is another hypothesis that has been proposed to explain discrepancies. The reported intention to participate in the vote and the actual participation of survey respondents is generally higher than that of the general population. Some have argued that there is a tendency to over report having voted since voting is socially desirable. Others have proposed that respondents to surveys do in fact participate more in the vote for a number of reasons : there are more socially integrated or participation in a survey stimulates voting behavior. (Granberg and Holmberg, 1992; Traugott et Katosh, 1979, 1981). Though the existence of misreport has been documented in some situations (Traugott et Katosh, 1979, 1981), it is not clear if overreporting is proportionally distributed among party supporters (Marsh, 1985; Jowell, 1993; Curtice and Sparrow, 1997) or not (Traugott and Katosh, 1979, 1981; Presser and Traugott, 1992). Another area of research is non response. A long-term hypothesis proposed to explain 2

6 discrepancy between the polls and the vote is item non response i.e. non response to the voting intention questions. It has been hypothesized that those who refuse to answer and/or those who indicate that they don t know whom they will vote for, are more likely to be conservative. The hypothesis has been confirmed at least in the U.K. (Jowell et al., 1993; Curtice, 1997; Curtice and Sparrow, 1997). Survey non response may also influence estimates of vote outcome. It is customary to estimate the impact of survey non response, due to not-at-home and refusals, with data about hard to reach respondents and those living in households where previous refusal has been recorded. For this type of study to be carried out, it is necessary that a certain response rate be reached, i.e. that a substantial number of attempts at reaching phone numbers in the sampling frame and at refusal conversion be carried out. Studies have consistently found that harder to reach respondents have specific characteristics in terms of demography (Triplett, 1998) and political attitude, i.e. that conservative voters are harder to reach (Traugott, 1987; Lau, 1994; Curtice and Sparrow, 1997; Curtice, 1997; Bolstein, 1991). Furthermore, respondents who come from households where a refusal has been recorded also have specific characteristics i.e. they are more likely to be women (Triplett, 1998) and conservative (Curtice and Sparrow, 1997). These last authors indicate that the more unpopular the Conservative party, the stronger the propensity of conservative respondents to refuse to indicate their vote intention. The more substantial survey non response is, the more substantial the likely bias against conservative vote. It is possible however that the bias is related to social desirability, thus varying between countries and with historical periods. Since surveys that use quotas usually have lower response rates, they are more likely to under represent the conservative vote (Curtice and Sparrow, 1997). Lau (1994) found no relationship between the size of samples and the quality of estimation of the vote but reported a correlation with a proxy, the number of days in the field. Vachon, Durand and Blais (1999) found that, after controlling for the size of samples, a relationship still existed between efforts made in order to increase the representativeness of the samples, and therefore the response rates, and the mean error as well 3

7 as the variance in prediction of voting intentions. Estimates based on hard to reach respondents and on aggregate survey data are not sufficient. It is important to also examine non respondents as such, i.e. those who were never reached or convinced to cooperate. The data indicate that they are more likely to be non voters (Bolstein, 1991; Marsh, 1985; Granberg and Holmberg, 1992) or to vote more for the Republican party (Bolstein, 1991). The third area of research is related to sampling frames. In theory, sampling frames should permit to represent the whole population of electors. A number of issues have been raised in this area, related to data collection modes. In North America, surveys conducted by telephone have spread, becoming the standard way to conduct surveys of the general population. A number of reasons explain this situation, the most obvious being the good coverage of all households by telephone and the low density of population. Meanwhile in Europe, quotas related to age, sex and occupation have been widely used till recently and data collection is often conducted using personal interviews either at peoples home or at street corners. If we concentrate on surveys conducted by telephone, one of the first coverage problem is related to unlisted phone numbers. Households with unlisted phone numbers have specific characteristics : their members seem less likely to vote (Bolstein, 1991) and, in Great Britain, more likely to be supporters of the Labour party (Curtice, 1997). Random digit dialling based frames permit to deal with this problem, but since households with unlisted phone numbers are considered to be less cooperative (Drew, Choudhry and Hunter, 1988; Traugott, Groves and Lepkowski, 1987), pollsters have a tendency to rely on list based frames. New problems have appeared in recent years with the proliferation of households with multiple phone lines and phone numbers. These households could be over represented in the sampling frames, particularly those using RDD based frames. Triplett (1998) notes a phenomenon that could compensate for this higher probability of selection, namely the fact that these households 4

8 seem harder to reach. The impact of an increase in the population residing in collective households has to be assessed. Their absence among survey respondents has been mentioned by Converse and Traugott (1986). Finally, pollsters will have to face the problems associated with the increasing number of individuals who have only a portable phone. These phone numbers are not in the sampling frames right now and, if they were, these respondents might be very reluctant to pay the cost. Weighting is rarely mentioned in the literature as a possible culprit for discrepancies. Traugott (1987) reports that weighting procedures vary widely among pollsters. Jowell et al. (1993) mention that weighting may have an impact only if weight variables are correlated with the vote. In some instances, it is possible that such a situation occurs : Some minority groups-- Blacks or Hispanics in the United States, immigrants in various countries -- may be harder to reach and may vote differently. This paper relies on multiple sources of data to test hypotheses based on the three main areas of research that could explain the systematic discrepancy between the polls and the vote found in the Quebec 1998 general election. 2.0 Methodology Since the first hypothesis was that a substantial number of voters had moved during the last days of the campaign, the first step was to conduct a post election survey among pre election respondents in order to determine whether they had in fact voted and for whom. The cooperation of two pollsters, Createc and Crop, allowed us to conduct a survey of fifteen 5

9 hundred pre election respondents 1, using a non proportional stratified sample in order to over represent non disclosers and supporters of third parties. The preliminary results of this survey have been disclosed elsewhere (Durand & Blais, 1999; Durand, Blais & Vachon, 1999) and will be presented succinctly here. A second source of information comes from one pollster, Crop, who provided us with all the data from the surveys conducted during the electoral campaign, including the administrative basis of the surveys which provides detailed information (time of call, result of call, interviewer, etc.) on all the attempts that were made in order to reach a household and complete an interview. Since up to 25 attempts had been made to reach a phone number and up to two attempts to convert initial refusals into completed interviews, it is possible to compare the vote intention of those who were harder to reach and/or who had previously refused to answer to the survey with vote intentions in the rest of the sample. One additional source of information comes from the published polls of the campaign (17 polls and six pollsters). We could also rely on two studies (Statmedia) conducted in Quebec in June 1997 and June 1998 on the use and listing of telephone lines. Furthermore, Crop provided us with the results of some of the tests on unlisted phone numbers that it carried out in the months following the election. Finally, we undertook a study to estimate the vote of collective households in a sample of constituencies. For each selected constituency, we asked the MP s personnel to identify the collective households present in their constituency and the voting section in which they were located. Information was gathered on these collective households (number of residents, proportion with a private telephone line, proportion of registered voters, likely proportion of 1 Only French-speaking respondents were interviewed because it is in that group (which represents 85% of the electorate) that movement of opinion was assumed to have taken place. Non French-speaking voters overwhelmingly support the Liberal party. 6

10 voters) as well as on the outcome of the election in the constituencies where these households were located. 3.0 The first hypothesis : the electorate moved The first easy explanation for the discrepancy between pollsters estimation of the vote and the actual outcome is that the electorate moved: Polls had not been conducted late enough in the campaign and voters changed their minds or decided to stay home. This interpretation can be tested in two different ways. The first is the post election poll conducted during the week following the election. This poll has shown that there was no late campaign swing in favor of the Quebec Liberal Party. In fact, there had been movement 2 between all political parties during the last week of the campaign but the net effect of these movements was slightly in favor of the Parti Québécois. Second, supporters of the Parti Québécois voted in greater proportion than supporters of the other two parties, the Liberal Party and Action Démocratique du Quebec (ADQ), a third party that finally got 12 p. cent of the vote (14% among French-speaking voters). As a consequence, as shown in Table 1, the overestimation of the Parti Québécois vote is even more substantial in the post election than in the pre election polls. Insert table 1 A second source of information on this issue is provided by the polls published during the campaign. These polls permit a time-series analysis of the evolution of vote intentions. A 2 Thirteen p. cent of those who declared both their vote intention and their actual vote changed their mind between the time of the pre-election survey and election day. These figures may be compared with figures of 5%, 7% and 10% for three different surveys presented by Jowell et al. (1993). 7

11 number of possible models were tested in order to estimate the model that best represents the results from the various surveys and gives a good forecast of the actual vote. The model that performs the best is one of stable vote intention throughout the four-week campaign, except for a small increase in support for the ADQ after the televised debate held at mid-campaign. These two tests lead to the same conclusion and confirm previous research (Jowell et al, 1993; Curtice, 199; Erikson and Wlezien, 1999): no late campaign swing occurred. 4.0 Was it non response? Two types of non response are examined in order to trace a possible impact on estimates of vote intention. Item non response refers to the fact that some respondents do not indicate which party they intend to vote for. Survey non response refers to the fact that some individuals are not interviewed either because they are not at home (or do not answer) when the survey firm calls or because they refuse to answer the survey. 4.1 Item non response The first question here is whether non disclosers, i.e., those who answer surveys but either say that they don t know whom they will vote for or refuse to provide the information, vote differently from disclosers. Insert table 2 A first source of information is the post election survey: How did non disclosers to the pre election polls vote? The results, presented in Table 2, show that among those who had said 8

12 they did not know or had refused to tell how they would vote, an equal proportion voted for the two main parties. Item non response must therefore be ruled out as the source of the discrepancy between the polls and the outcome since Quebec pollsters allocate 60% of non disclosers to the Liberal party, 30% to the Parti Québécois and 10% to ADQ. The post election polls suggest that this allocation may be slightly too generous for the Liberal party. It is not because of the non disclosers that the Liberal vote was underestimated in the polls. Another source of information leads to the same verdict. Since pollsters ask a leaning question 3 to those who do not divulge their preference in the initial vote intention question, it is interesting to examine the answers given by the leaners. The data base from Crop comprising three regular surveys conducted during the campaign shows that the proportion of non disclosers to the first question was stable throughout the campaign (as can be seen in Table 3): the proportion of don t knows varies from 9 to 12 p. cent, the proportion of refusals from two to four p. cent and the proportion of those who say they will not vote or will spoil their ballot is two to three p. cent. Insert table 3 Table 3 shows that, among initial non disclosers, 45 to 49 p. cent maintained in the follow up leaning question that they did not know whom they would vote for, this even in the last poll of the campaign, that from 14 to 20 p. cent refused to declare their intention and from nine to 12 p. cent said that they would not vote or would spoil their ballot. Only 26 p. cent of non disclosers to the first question indicated which party they were inclined to support when asked the leaning question. These 26 p. cent turn out to be slightly more favorable to the Liberal Party than those who revealed their vote intention in the initial question. 3 The leaning question was put to all those who responded to the first question on vote intention that they did not know how they would vote, that they would not vote or that they would spoil their ballot as well as to those who refused to answer. These respondents were asked which party they were leaning toward. 9

13 In view of this information, it is possible to conclude that the second hypothesis related to the vote of non disclosers is validated. Previous research (Jowell et al. 1993; Curtice, 1997; Curtice and Sparrow, 1997) is confirmed: Non disclosers are slightly more inclined to vote a more conservative party, in this case the Quebec Liberal Party,. However, this is taken into account by Quebec pollsters who attribute 60% of the vote of non disclosers to the Quebec Liberal party. The culprit has to be elsewhere. 4.2 Survey non response Survey non response comprises two components: a) households or individuals who cannot be reached and b) households or individuals who refuse to answer the survey Hard to reach households and individuals Some pollsters and academics, in concordance with consistent findings of survey research (Traugott, 1987; Lau, 1994; Curtice and Sparrow, 1997; Curtice, 1997; Bolstein, 1991), have proposed the hypothesis that hard to reach individuals are more likely to support the Quebec Liberal party. Our analysis, presented in tables 4 and 5, shows no relation between the number of calls necessary to reach a household or to complete an interview and answers to the vote intention question. All the tests are highly non significant, this despite the fact that the total number of respondents is more than three thousands and up to 25 calls had been made in order to complete interviews. Insert tables 4 and 5 This result is surprising in view of previous research and given the fact that there is a 10

14 relationship between the number of calls and socio-demographic characteristics that are linked to vote intention. In this study, the number of calls necessary to reach a phone number or to complete an interview is higher among younger respondents, among full-time workers and students and among residents of Montreal suburbs. The number of calls necessary to complete an interview is also higher in the Montreal region, among the better educated and non French speakers. It is possible that the diverse characteristics of hard to reach individuals counterbalance each other and that the various impacts on voting intentions cancel out Non cooperative households or individuals As with hard to reach respondents, it is possible to ascertain the potential bias associated with refusals by examining the vote intention of less cooperative respondents. Close to 12 p. cent of respondents in the Crop data base initially refused to give an interview or belong to a household where an interview had been refused. As can be seen in Table 6, respondents/households where a refusal had taken place tended to refuse to reveal their vote intention. However, there is also a tendency for supporters of the Liberal Party to come from households where a refusal had been recorded. Liberal voters were not, however, more numerous among those who had personally refused to be interviewed and among those who had refused more than once. Insert table 6 A similar pattern emerges out of the post election survey: Those who refused to indicate how they had voted were more numerous in households where a refusal had been recorded (in the pre election poll). Furthermore, respondents from these households are more likely to have 11

15 voted for the Liberal Party (43% vs. 28% in the whole sample) while respondents who had personally refused to answer the survey were more likely to have voted for the Parti Québécois. Because the former group is twice as large as the latter, the most substantial bias is an under representation of the Liberal party. These results confirm previous research by Curtice and Sparrow (1997). Insert table Non respondents It is also possible that non respondents (never reached or not converted refusals) differ from respondents who were hard to reach or who initially refused to answer but finally accepted. If we look at the relationship between language spoken at home and vote intentions, we get further clues as to the characteristics of non respondents. It appears that the vote intentions of the non French-speaking respondents that have been reached by Crop do not reflect the standard estimation made by political scientists. It is generally estimated that close to 90 p. cent non French-speaking Quebeckers vote for the Liberal Party. Table 8 shows that, in the actual data base comprising the three surveys, 77 p. cent of English-speaking respondents and 64 p. cent of non French/non English-speaking respondents (respectively 85% and 71% of those who indicate their intention) intend to vote for the Liberal Party. Insert table 8 Since there is a differential response rate according to language spoken at home, non French speaking respondents being 13 p. cent of the final sample while they are 17 p. cent of the general population according to the Canada 1996 Census, the consequence of adjustment 12

16 weighting is to increase the bias against the Liberal party. We may therefore conclude that survey non response may be partly responsible for discrepancies in estimation of vote intention, non respondents being more likely to favor the Liberal party. 5.0 Sampling frame issues Three types of bias have been identified, some specific to Quebec pollsters, others generic, at least in North America and in most industrialized countries. Sampling frames used by most Quebec pollsters during the electoral campaign did not include unlisted telephone numbers. The two other biases concern households with more than one listed telephone number and collective households. We examine the possible impact of these biases. 5.1 Unlisted telephone numbers The Statmedia 1998 study has shown that the proportion of respondents who indicate that their telephone number is not listed or who are not sure whether it is listed is 12 p. cent in Quebec, 17 p. cent in the Montreal region. This information, presented in table 9, is confirmed by four Crop surveys conducted during the months following the election. Both sources conclude that non listed telephone numbers are more likely to belong to non French speaking Quebeckers and to younger people (1ess than 25 years old). This could explain in part the discrepancy between Census Canada and the actual estimation of the proportion of non French speaking and young people in the samples (see Table 11). 13

17 Insert table 9 Crop provided us with the results of the surveys it conducted in February, April and August It included unlisted phone numbers 4 in its sample in order to evaluate the possible impact of the former exclusion. Crop also asked respondents whether their telephone number was listed in the directory. From 16% to 22% of respondents came from unlisted phone numbers and 11% indicated that their telephone number was not listed or that they did not know if it was. In two of the three polls, unlisted respondents appeared more likely to intend to vote for the Liberal Party and less likely to favor the Parti Québécois. Furthermore, respondents who indicated that their telephone number is not listed are less likely to favor the Parti Québécois. We can thus conclude that in the Quebec case, contrary to England (Curtice, 1997), excluding unlisted phone numbers has contributed to the bias against the Quebec Liberal party. 5.2 Doubly listed phone numbers A recent phenomenon, even more prevalent with Internet access becoming more popular, is that a number of households may be reached at more than one telephone number. In some families, there is a telephone number for the kids and one for the parents. The Statmedia study carried out in 1997 showed that, at that time, close to 11 p. cent of the 3008 respondents to a survey on media consumption could be reached at more than one telephone number. Table 10 4 The RDD generated sample is originally divided into two parts, listed and unlisted. The unlisted part is composed of telephone numbers that were not found in the transcribed telephone directories. A majority of respondents coming from the unlisted part of the original sample indicated that their telephone number was listed. This situation may be attributed to the delay between the publication of the directory and its integration in the data base from which samples are generated. 14

18 shows that they were 15 p. cent among the more educated, 19 p. cent among 15 to 24 years old as opposed to four p. cent among the 65 years old and over, 20 p. cent among those who live in households where three or more members are 15 years old and over and 20% among those whose principal occupation is studying. Close to one of four households with a high income (more than 80,000$CAN) could be reached at more than one telephone number. Insert table 10 We may therefore conclude that some bias may be due to the higher selection rate in households with more than one telephone number, though it is not possible to specify the impact of this bias on estimates of vote intention. 5.3 Collective households Changes in the demographic composition of the population may also have played a role in the estimation of vote intention during the 1998 electoral campaign in Quebec. According to the Census, the Quebec population aged 65 years and older has raised from 13% of the total population over 18 in 1986 to 17% in About 10% of those over 65 live in collective households, -- eg. old age pensioners, physically disabled people, members of religious communities; their number increased by 25 p. cent from 1986 to People aged 65 years and older constituted 71% of the people living in collective households in These people are included in the sampling frames to a certain degree, i.e., when they have a private telephone line. We have tried to estimate the vote of collective households using data from the Census and information gathered from a sample of constituencies (one out of 10). 15

19 It is estimated that about 48% of the people living in collective households have access to a private telephone number and may therefore be reached by survey companies; the proportion registered to vote is estimated at 70% and the overall proportion of voters at about 43%. A conservative evaluation for 1998 gives an estimate of 52,000 voters from collective households, 1.3% of all voters. Insert Figure 1. These voters differ in their voting behavior from other electors in the same constituencies. Figure 1. shows that, in the polling sections where more than 40% of registered electors are residents of collective households, the participation rate is 11 percentage points lower than in the other polling sections. In these same sections, the proportion who vote for the Liberal Party is 20 percentage points higher. Even though half of the voters living in collective households may be reached by pollsters, adjustment weighting used by these companies does not take into account this segment of the voting population since it is based on Statistics Canada' s Census of private households. 5.4 Sampling frames and weighting All the biases related to the sampling frame tend to over represent Parti Québécois supporters and to under represent supporters of the Liberal Party. Can these biases be corrected by adjustment weighting on the basis of Census data? Adjustment weighting based on the Census has two flaws. First, Census data may be somewhat outdated and second, the population of voters does not distribute like the general population: for example, younger people are harder to reach but they are also less likely to vote than older people. Therefore, weighting according to age groups may lead to over 16

20 represent people who will not vote. Some immigrants included in the Census do not have the right to vote. Members of collective households have the right to vote and do vote but they are not included in the Census data of private households used by pollsters. Furthermore, we have shown that respondents may differ from non respondents of the same age/language/sex group. Does weighting based on probability of selection and adjustment for differential response rates provide a more accurate description of vote intentions? The short answer is no as shown in Table 11. It does not improve the estimation of vote intentions, the discrepancy being even slightly larger than with adjustment weighting. Insert table Conclusion In this study, we have examined the possible explanations for the discrepancy between estimates of vote intentions and the actual outcome of the 1998 Quebec election in which a systematic bias in favor of the Parti Québécois had appeared in the polls. We have shown that the gap cannot be imputed to late campaign shift in vote intentions nor to an inadequate allocation of the vote of non disclosers. The sources of the discrepancy appears to be survey non response and sampling frames. Most of the time the biases appear to be relatively small but their overall effect is substantial because they are all in the same direction. The study indicates that pollsters will have to devote greater effort in improving the quality of their sampling frame and weighting procedures and in increasing their response rate, more specifically among the non French-speaking population, in order to come up with more reliable 17

21 measures of party support. It should be pointed out that the problems we have identified are not specific to Quebec. Similar situations have occurred in a number of other elections, with usually the same direction of bias : the under representation of more conservative vote. It is perplexing to note that the problem occurred despite the fact that the methodology used by Quebec pollsters is quite orthodox : Random samples with call backs are used and response rates are generally around 60%. 18

22 References Bishoping K. and Schuman H. (1994). Pens and Polls in Nicaragua : An analysis of the 1990 Preelection Surveys. American Journal of Political Science, 36 (2), Bolstein, R. (1991). Comparison of the likelihood to vote among preelection poll respondents and nonrespondents. Public Opinion Quarterly, 55, Converse, Ph. E. and Traugott, M. W. (1986). Assessing the Accuracy of Polls and Surveys, Science, 234, Curtice, J. (1997). So How Well Did They Do? The Polls in the 1997 Election. Journal of the Market Research Society, 39 (3), Curtice, J. and Sparrow, N. (1997). How Accurate Are Traditional Quota Opinion Polls? Journal of the Market Research Society, 39 (3), Drew, J. D., Choudhry, G. H. And Hunter, L. A. (1988) Nonresponse issues in government telephone surveys. In Groves, R. M., Biemer, P. P., Lyberg, L. E., Massey, J.T., Nicholls!!, W. L. and Waksberg, J. Telephone Survey Methodology, New York: Wiley, Durand, C. and A. Blais (1999). Why did the polls go wrong in the 1998 Quebec election : the answer from post election polls, BMS, 62, Durand, C., Blais, A. and S. Vachon (1999). Why did the polls go wrong in the 1998 Quebec election? Paper presented at the 54 th Annual Conference of the American Association for Public Opinion Research (AAPOR), St. Pete Beach, Florida, May 13-16,

23 Erikson, R. E. and Wlezien, C. (1999). Presidential Polls as Time Series : The case of 1996, Public Opinion Quarterly, 63(2), Granberg, D and Holmberg, S. (1992) The Hawthorne Effect in Election Studies : the Impact of Survey Participation on Voting. British Journal of Political Science, 22 (2), Howell, S.E. and Sims, R. T. (1994). Survey research and racially charged elections : The case of David Duke in Louisiana. Political Behavior, 16 (2), Jowell, R., Hedges, B., Lynn, P., Farrant, G. and Heath, A. (1993). The Polls - a Review; The 1992 British Election : the Failure of the Polls. Public Opinion Quarterly, 57, Lau, R.R. (1994). An Analysis of the Accuracy of "Trial Heat" Polls During The 1992 Presidential Election. Public Opinion Quarterly, 58, Marsh, C. (1985). Prediction of Voting Behavior from a Pre-election Survey, Political studies, 33(4), Presser, S. and Traugott, M. W. (1992). Little White Lies and Social Science Models : Correlated Response Errors in a Panel Study of Voting, Public Opinion Quarterly, 56 (1), Traugott, M. W. (1987). The importance of Persistence in Respondent Selection for Pre- Election Surveys, Public Opinion Quarterly, 51 (1), Traugott, M. W., Groves, R. M. and Lepkowski, J. M. (1987). Using Dural Frame Designs to Reduce Nonresponse in Telephone Surveys, Public Opinion Quarterly, 51 (4),

24 Traugott, M. W. and Katosh, J. P. (1981). The Consequences of Validated and Self-Reported Voting Measures, Public Opinion Quarterly, 45, Traugott, M. W. and Katosh, J. P. (1979). Response Validity in Surveys of Voting Behavior, Public Opinion Quarterly, 43 (3), Traugott, M. W. and Price, V. (1992). The polls - a review. Exit polls in the 1989 Virginia gubernatorial race : where did they go wrong? Public Opinion Quarterly, 56, Triplett, T. (1998). What is Gained from Additional Call Attempts and Refusal Conversion and What are the Cost Implications? Survey Research Center, University of Maryland, October Vachon, S., Durand, C. and Blais, A. (1999). Les sondages moins rigoureux sont-ils moins fiables? Canadian Public Policy/Analyse de politiques, 25 (4), in press. 21

25 Table 1 Vote intention, reported vote and election results among French-speaking respondents Estimates of voter intention and actual vote N P.Q. Lib. ADQ Other parties (+cancel) non disclosers will not vote - will cancel/ Did not vote Pre election vote intention Post election reported vote % 30% 16% 1% (11%) attrib % 31% 13% 2% (8%) attrib. (7%) withdrawn (12%) withdrawn election (estimation of Frenchspeaking voters) 50% 35% 14% 1% (22%) 22

26 TABLE 2. Reported vote of non disclosers to the pre-election poll Reported vote P.Q. Lib. ADQ Other parties +cancel refusals Did not vote Will cancel/will not vote (N=105) undecided (N=108) refusal (N=188) 9% 7% 4% 3% 13% 65% 22% 27% 11% 6% 21% 13% 26% 21% 10% 3% 37% 3% 23

27 Table 3 Response to the initial vote intention question and to the leaning question (CROP pre election surveys) Oct 30 - Nov 4 Nov 6-11 Nov Vote intention Leaning Vote intention Leaning Vote intention Leaning Count % Count % Count % Count % Count % Count % ADQ 293 5,4% 28 4,1% 277 5,2% 10 1,1% ,7% 56 5,9% Lib ,0% 74 10,8% ,1% 99 11,1% ,4% 90 9,5% PQ ,6% 75 11,0% ,8% ,5% ,9% 82 8,7% Other parties 119 2,2% 20 2,9% 74 1,4% 0,0% 75 1,4% 8,9% Will cancel, will not vote 108 2,0% 79 11,5% 118 2,2% 80 9,0% 157 2,9% ,9% Undecided 466 8,7% ,9% ,7% ,1% ,0% ,4% Refusal 112 2,1% 88 12,8% 144 2,7% ,2% 199 3,7% ,7% Total ,0% ,0% ,0% ,0% ,0% ,0% Note: The N s correspond to the weighted number of individuals represented in the population divided by 1,

28 Table 4 Number of calls necessary to reach a household and vote intention (first + leaning questions combined) one two three four to six seven and more Count % Count % Count % Count % Count % ADQ 660 7,9% 217 7,5% 108 6,1% 216 8,8% 38 5,7% Lib ,7% ,5% ,2% ,9% ,0% P.Q ,7% ,7% ,8% ,5% ,6% Other parties Will cancel, not vote 171 2,1% 57 2,0% 33 1,9% 17,7% 19 2,9% 111 1,3% 20,7% 66 3,7% 54 2,2% 11 1,7% Undecided 570 6,9% 208 7,1% 160 9,0% 201 8,2% 39 5,9% Refusal 204 2,5% 46 1,6% 61 3,4% 93 3,8% 14 2,2% Total ,0% ,0% ,0% ,0% ,0% Note: The N are weighted according to the 1996 Census and then divided by 1,000. Since three surveys are combined the numbers correspond to three times the population (divided by 1,000). 25

29 Table 5 Number of calls necessary to complete an interview and vote intention (first + leaning questions combined) one two three four or five six to nine ten and more Count % Count % Count % Count % Count % Count % ADQ 394 8,5% 263 7,8% 138 5,2% 294 9,7% 125 6,1% 24 6,1% Lib ,8% ,5% ,1% ,7% ,4% ,1% PQ ,1% ,5% ,1% ,5% ,8% ,5% Other parties Will cancel, not vote 52 1,1% 72 2,2% 69 2,6% 41 1,4% 50 2,4% 12 3,1% 64 1,4% 53 1,6% 44 1,7% 45 1,5% 48 2,3% 9 2,2% Undecided 303 6,5% 220 6,5% 252 9,6% 233 7,7% 142 6,9% 29 7,3% Refusal 79 1,7% 64 1,9% 72 2,7% 110 3,6% 84 4,1% 11 2,7% Total ,0% ,0% ,0% ,0% ,0% ,0% Note: The N are weighted according to the 1996 Census and then divided by 1,000. Since three surveys are combined the numbers correspond to three times the population (divided by 1,000). 26

30 Table 6 Initial refusal to answer the survey and vote intention (first + leaning questions) No refusal One household refusal converted One selected respondent refusal converted Two refusals (household or respondent) converted Count % Count % Count % Count % ADQ ,8% 74 7,9% 36 8,2% 9 2,1% Lib ,2% ,6% ,9% ,4% PQ ,1% ,9% ,8% ,1% Other parties Will cancel, not vote 260 1,8% 21 2,2% 8 1,7% 8 1,9% 243 1,7% 15 1,6% 4 1,0% 0,0% Undecided ,2% 67 7,1% 32 7,4% 50 11,1% Refusal 312 2,2% 44 4,7% 48 11,0% 15 3,4% Total ,0% ,0% ,0% ,0% Note: The N are weighted according to the 1996 Census and then divided by 1,000. Since three surveys are combined the numbers correspond to three times the population (divided by 1,000). 27

31 Table 7 Initial refusal and reported vote (sub sample from post election survey) No refusal One household refusal converted one selected respondent refusal converted two successive refusals converted Count % Count % Count % Count % ADQ ,0% 4 4,0% 0,0% 8 26,3% Lib ,7% 36 35,4% 12 18,6% 4 13,6% PQ ,2% 13 13,1% 21 33,0% 9 30,0% Other parties, cancel 39 1,8% 4 4,1% 0,0% 4 13,6% Refusal 152 7,0% 36 35,5% 10 16,1% 5 16,5% Did not vote ,3% 8 7,9% 21 32,3% 0,0% Total ,0% ,0% ,0% ,0% Note: the weighting is based on pre-election weights (Census divided by 1,000). 28

32 Table 8 Vote intention and language spoken at home French-speaking English-speaking Other language Count % Count % Count % ADQ ,0% 17,8% 32 4,2% Lib ,2% ,3% ,3% PQ ,5% 180 8,5% ,6% Other parties Will cancel, not vote 177 1,3% 99 4,7% 20 2,7% 240 1,8% 15,7% 8 1,0% Undecided 999 7,5% 132 6,3% 47 6,2% Refusal 354 2,7% 35 1,7% 30 3,9% Total ,0% ,0% ,0% Note: The N are weighted according to the 1996 Census and then divided by 1,000. Since three surveys are combined the numbers correspond to three times the population (divided by 1,000). 29

33 Table 9 Characteristics of respondents who declare that their telephone number is unlisted or who don t know whether it is listed (Statmedia, 1998) Montreal urban area 17% Montreal suburbs 15% English-speaking 14% Other language 18% years old 18% Total 12% 30

34 Table 10 Characteristics of respondents living in households with more than one telephone number (Statmedia, 1997) More than one phone number Total 11% college educated 15% years old 19% 65 years old and over 4% three or more 15 years old and over in household Household income (80,000$Can and more) Individual income (60,000$Can and more) 20% 25% 23% 31

35 Table 11 Characteristics of the Crop sample, unweighted, adjusted and weighted according to inverse of selection and response rate Non weighted Adjusted w. Census data (,000 X3) Weighted by inverse of selection/resp. Rates (,000 X3) N % N % N % Vote intention: ADQ PLQ PQ Other parties Will cancel, not vote Undecided Refuse to say Total ,9 33,7 44,5 1,7 1,7 7,7 2,8 100, ,7 35,5 43,5 1,9 1,6 7,3 2,6 100, ,0 33,7 44,7 1,8 1,6 7,7 2,6 100,0 Age groups years years years years years 65 years ,4 17,9 25,5 19,5 11,6 16, ,1 21,1 23,0 17,9 11,2 15, ,6 18,1 24,9 20,6 10,6 13,3 Sex Men Women ,5 55, ,4 51, ,0 54,0 Language spoken at home French English Other language ,2 9,6 3, ,2 13,1 4, ,0 9,1 3,8 32

36 33