EXPERTS POLITICAL PREFERENCES AND THEIR IMPACT ON IDEOLOGICAL BIAS

PPQ341051 1/9/09 11:48 pm Page 1 PARTY POLITICS VOL 15. No.? pp. 1 23 Copyright 2009 SAGE Publications Los Angeles London New Delhi www.sagepublications.com Singapore Washington DC EXPERTS POLITICAL PREFERENCES AND THEIR IMPACT ON IDEOLOGICAL BIAS An Unfolding Analysis based on a Benoit-Laver Expert Survey Luigi Curini ABSTRACT Expert surveys have become increasingly popular among political scientists. One of the problems of using surveys (of any sort) to estimate party positions is that respondents can be influenced by their subjective political views. As a consequence, experts may give biased responses, and such (ideological) bias may affect certain parties more than others. In this paper, we use the latest expert survey of Benoit and Laver (2006) to unfold the ideal points of the respondents to the survey. By employing the estimated ideal points, we show that in almost 16 percent of the cases analyzed, there is evidence of an ideological bias in the experts placements of parties along the left right dimension, especially among right-wing parties (but not necessarily extreme-right parties). We examine two methods designed to generate less biased estimates. The first one is directly based on a regression technique, while the second is based on the negligibility of ideological bias in experts answers to more specific policy questions. The paper concludes by examining the consequences of these findings for empirical research. KEY WORDS expert survey ideological bias Italy left right dimension unfolding analysis Since Castles and Mair (1984), expert surveys have become increasingly popular as a way of identifying the policy preferences of political parties. In such cases, a survey of country specialists is directly conducted, whereby they are asked to locate political parties in their own countries on a set of predefined policy dimensions: from a general left right scale, to a variety of 1354-0688[DOI: 10.1177/1354068809341051]

PPQ341051 1/9/09 11:48 pm Page 2 PARTY POLITICS more specific policy dimensions. Several factors help to explain the success of expert surveys (Mair, 2001). First of all, such surveys provide information on party policy positions in a common, standardized format across a wide range of countries. Second, the fact that they reflect the judgement of experts who are presumably well-informed gives them weight and legitimacy. Finally, expert surveys are quickly and easily compared to other forms of analysis (such as the analysis of the content of party electoral programs, or legislative behavioral studies). In the present work we are going to analyze the expert survey conducted by Benoit and Laver (2006), which is probably the most comprehensive and ambitious of its kind. We will be focusing in particular on the Italian case, given that in this case we can avail ourselves of two expert surveys, the one conducted soon after the other (2003 and 2006), using the same methodology. This fact can, inter alia, allow us to check for the robustness of our results. However, instead of using experts responses to identify the policy preferences of parties, we will use these answers to directly map the ideal points of the respondents. Our interest goes beyond the merely descriptive. One of the problems of using surveys of any kind to estimate party positions is that respondents may be influenced by their own subjective political views. As a consequence, there is the risk that experts can give biased responses and that such (ideological) bias affects certain parties more than others. By using the estimated ideal points of the said respondents, we are going to show that there is an ideological bias in the Benoit and Laver survey, particularly in terms of the left right dimension. In this case, approximately 16 percent of those parties analyzed appear to be significantly affected by it. As a result, the expert survey is incapable of directly producing a valid position for such parties. Thus, we shall be proposing two different ways of generating less biased scores. We shall then go on to discuss the importance of these findings for empirical research. The Benoit-Laver Expert Survey and the Italian Case Benoit and Laver (hereafter BL ) conducted their survey for the main part in 2003: it covered 47 countries (all European countries plus Canada, United States, Australia, New Zealand and Japan), resulting in 1491 expert opinions, giving it a far more comprehensive coverage than any other survey of its kind. The typical expert in their particular survey was an academic specializing in the political parties and electoral politics of his/her own country. With regard to parties policy preferences, BL deployed a core set of four substantive policy dimensions in every national survey in order to allow for the direct comparison of policies between the various countries in question. Furthermore, they also measured policy positions on more country-specific issues judged on a country-by-country basis. The said four core dimensions were: economic policy, social policy, environmental policy and the decentralization 2

PPQ341051 1/9/09 11:48 pm Page 3 CURINI: EXPERTS POLITICAL PREFERENCES of decision making. For each policy dimension, BL used a scale from 1 to 20, with the lower numbers indicating a left-wing position, and the higher numbers indicating a traditionally right-wing position. To give but two examples, the previous economic policy was defined as (1) Promoting the raising of taxes to finance increased spending on public services and (20) Promoting the reduction of public services to cut taxes ; the social policy dimension, on the other hand, was defined as (1) Favouring liberal policies on matters such as abortion, homosexuality, and euthanasia and (20) Opposing liberal policies on matters such as abortion, homosexuality, and euthanasia. A party that is pro-market and liberal on social issues would be given a score of around 20 on economic policy and around 1 on social policy. The first Italian expert survey (the only one mentioned in BL s study) was conducted in 2003. Total respondents numbered 54 (only Germany, Sweden and Britain had a higher number of respondents among those Western European nations involved), with a response rate of 30% (two points higher than the mean response rate for the entire sample). Benoit and Laver repeated their Italian survey three years later, at the time of the 2006 Italian general election. 1 In this second case, there was a slight reduction in the number of respondents (down to 40). Given the anonymity of the respondents, we cannot directly check to see whether there is any overlap between the 2003 and the 2006 sets of respondents. However, the fact that we can avail ourselves of two surveys conducted in the same country is particularly useful since it allows us to see whether there is any temporal trend in the distribution of the political experts ideal points. This, as we shall see, can even be used as an indirect check on the robustness of our results. In addition to the four dimensions mentioned earlier, both Italian surveys cover five other policy dimensions, namely: deregulation; immigration; EU policy authority (the question of whether the scope of the EU s policy decisions should be expanded or restricted); EU accountability (the question of whether EU institutions should provide direct links to citizens through representative institutions such as the European Parliament, or whether they should be controlled by national governments instead); and the question of the expansion of the EU s role in collective security, peacekeeping and other military affairs. The experts were also asked to locate each party along a general left right scale, having taken all aspects of party policy into account. Finally, and somewhat unusually for an expert survey, BL followed the Laver Hunt (1992) method and asked the experts to rate their closeness to each of the political parties on a scale of 1 to 20 through a sympathy question, taking account once again of all aspects of party policy (whereby a score of 1 means that the expert in question identifies very much with the said party s policy preferences). The aim of the latter procedure was to test for any possible respondent bias, by checking whether an expert s placing of parties on substantive dimensions was in some way correlated with his/ her personal sympathy with that party s policies. The sympathy question, however, may even be used to locate respondents ideal points in relation 3

PPQ341051 1/9/09 11:48 pm Page 4 PARTY POLITICS to their own policy preferences. To this end, the use of an unfolding analysis is an attractive methodological solution, and the following section is going to examine this option. For the time being, Table 1 shows parties scores for each policy dimension resulting from the Italian 2006 survey (the corresponding Table for 2003 can be found in BL), while Figure 1 plots the mean left right position as well as the sympathy score for each party taken into consideration in the two surveys. Interestingly, by excluding the three more left-wing parties (RC, PCDI and Greens) we can produce an almost perfect linear relationship between the left right placement of a party and its average level of sympathy. Unfolding the Policy Preferences of Italian Political Experts An unfolding model is a model for preferential choice, i.e. it assumes that different individuals perceive various objects of choice (or set of stimuli, to use psychometric jargon) in the same way, but differ over what they consider an ideal combination of the object s attributes. In the unfolding method, the data are usually the preference scores (such as rank-orders of preference) of different individuals in relation to a chosen set of objects. These data may be conceived as proximities between the elements in two sets: individuals and chosen objects. Individuals are represented as ideal points within a geometric space, so that the distance from each ideal point to the object point corresponds to the preference score. This geometric space may be unidimensional if subjects use a single criterion to evaluate the stimuli. In this event, a single dimension would be appropriate to the arraying of the subjects (and eventually of the stimulus points). Otherwise, should subjects employ several different criteria in order to form their judgements, then a multidimensional space may be needed (Jacoby, 1991). The stronger a respondent s preference for an object, the smaller the distance from the ideal point to that object point, and vice versa. In this sense, the aim of unfolding analysis is to estimate the locations of each ideal point, using the respondent s expressed preferences for the objects in question (Borg and Groenen, 2005). In our case the said objects are political parties. There are two main categories of unfolding model. The first, that of external unfolding models, assumes that a similarity configuration of the chosen objects is already given. In this case, we are only going to try to locate the respondents ideal points. In the second category on the other hand, that of internal unfolding models, only preference data are given, from which both object configuration and ideal points have to be derived. We extracted information from the BL survey about parties locations, in order to then estimate an external unfolding model under two different scenarios. 2 In the first scenario, we assume that the respondents judge their closeness to each party mainly in terms of the left right dimension, which 4

PPQ341051 1/9/09 11:48 pm Page 5 CURINI: EXPERTS POLITICAL PREFERENCES Table 1. Summary data from the Italian expert survey (2006) EU EU EU Economic Social Environment Decentralization peacekeeping Immigration Deregulation accountability authority Left right Rc 3.57 3.34 5.25 13.74 16.89 2.85 3.13 7.26 11.38 2.28 Pdci 3.56 4.09 6.09 13.38 15.94 3.37 3.59 7.53 10.53 3.05 Green 5.44 3.28 2.30 11.30 15.90 3.32 5.53 6.56 7.65 4.15 Ds 6.39 5.78 7.97 9.47 8.00 4.90 7.64 6.36 5.68 6.10 Rose 11.75 1.94 10.41 8.70 5.70 6.43 14.29 5.63 6.45 8.19 Marg 8.97 12.59 9.14 9.71 7.23 7.03 10.17 7.21 6.16 8.51 Iv 8.52 9.04 9.00 10.86 8.06 8.79 8.58 7.33 7.62 9.35 Udeur 9.16 16.36 11.89 11.72 7.33 10.18 9.25 10.14 9.91 10.60 Pri 13.64 8.18 14.70 9.14 5.62 11.53 14.09 8.71 8.88 11.64 Npsi 11.26 6.68 11.80 9.45 6.63 10.11 11.64 9.75 9.58 11.82 Udc 10.67 17.68 12.71 10.77 6.97 11.59 10.43 10.72 9.47 12.33 Fi 16.82 13.84 17.32 8.03 6.39 14.66 15.74 15.30 15.74 14.98 An 10.41 17.68 14.42 13.62 6.00 16.43 8.62 14.65 13.53 16.28 Ln 16.39 18.38 16.26 2.11 12.19 19.33 14.49 16.16 18.76 17.30 Msft 7.78 18.96 12.40 17.91 11.56 19.14 5.40 18.19 17.64 19.03 As 7.81 18.76 12.82 17.67 10.47 19.37 5.52 18.06 17.75 19.08 Note: Rc = Communist Refoundation Party; Pci = Italia Communist Party; Pds = Democratic Party of the Left; Ds = Democrats of the Left; Psi = Italian Socialist Party; Psdi = Italian Social Democratic Party; Psu = United Socialists Party; Pri = Italian Republican Party; Dc = Christian Democracy; Ppi = Italian Popular Party; Pli = Italian Liberal Party; Pr = Radical Party; Ccd = Christian Democratic Centre; Ln = Northern League; Msi = Italian Social Movement; An = National Alliance; Fi = Go Italy. 5

PPQ341051 1/9/09 11:48 pm Page 6 PARTY POLITICS Figure 1. Relationship between the left right score of a party and its average level of sympathy (expert survey: Italy 2003 and 2006). is considered to be the dimension that constrains parties positions across a broad range of policies to the greatest degree (Gabel and Huber, 2000). However, the left right division may be too rough to take account of several important political divisions. For example, a classical liberal, who supports both liberal social policies and laissez-faire economics has no clearly defined place on a left right scale. As a result, in our second scenario we have assumed that the experts judge their proximity to each party in terms of two distinct dimensions (in theory at least): the Economic dimension (i.e. taxes vs. spending) and the Social dimension. 3 This pair was chosen for two reasons: on the one hand, they remain a classical source of structure in political competition and in the definition of a person s ideological position; 4 on the other hand, and from a more pragmatic point of view, they allow us to pursue a more informed comparative analysis, given that the Economic and Social dimensions are two of the four dimensions deployed in each of the surveyed countries, as we have already pointed out. The main alternative to the method employed here would have been to run a multidimensional scaling analysis (MDS) on the sympathy question in order to recover the positions of the object points directly from the data, and then to estimate the external unfolding analysis using the object points locations revealed by the MDS (this strategy is sometimes called successive unfolding in the literature: Jacoby, 1991). This method would have provided us with direct empirical evidence of respondents perceptual structures and the evaluative criteria they actually employ when thinking about the objects in question. However, the locations of the coordinate axes (i.e. the dimen- 6

PPQ341051 1/9/09 11:48 pm Page 7 CURINI: EXPERTS POLITICAL PREFERENCES sions of the space in which the respondents ideal points are located) are completely arbitrary in this case. In other words, the axes are simply a device to hang the point within the m-dimensional space, without furnishing any intrinsic substantive interpretation. It is the researcher s duty to label the dimensions differentiating the objects (Poole, 2005). On the contrary, by constraining the objects point locations (i.e. the parties) on a set of policy dimensions established ex-ante as we have done, we were able to explicitly define a common policy space in which to locate the respondents ideal points, thus enabling a cross-national comparison to be made. 5 In the Italian case, the fit measures of the unfolding analysis in the two scenarios are very similar and acceptable. 6 In other words, both the left right dimension and the two-dimensional space comprised by the Economic and Social dimensions are capable of effectively recovering respondents ideal points in relation to the objects locations. Figure 2 plots the locations of the Italian parties in the first scenario as well as the Kernel density distribution of the respondents ideal points in the two surveys. As we can see, the shape of the distribution in the two periods remains quite stable. 7 This is good news from the point of view of our analysis. Indeed, given the short time span separating the two Italian surveys, any marked difference between the two distributions would have made us rather doubtful about the reliability of our results. According to our estimation, roughly 90 percent of the respondents ideal points in 2003 (and 85% in 2006) were situated to the left of the Figure 2. The Kernel density distribution of the ideal points of the respondents along the left right dimension (Italy 2003 and 2006): 1 = left; 20 = right. Mean expert position reported with.95% c.i. 7

PPQ341051 1/9/09 11:48 pm Page 8 PARTY POLITICS midpoint of the left right scale (i.e. 10.5), thus giving a mean position in both surveys fairly close to the Democrats of the Left Party DS (mean score in 2003: 6.02 standard error (s.e.):.40; mean score in 2006: 6.42 s.e.:.51). The location of the experts ideal points under the second scenario, together with the experts mean location and the parties positions, are plotted in Figure 3. Interestingly, the quadrants that appear in this figure can be used to describe four broad systems of belief (see Maddox and Lielie, 1984). The southwest quadrant indicates a person of a Social Democrat persuasion, i.e. a person who supports state economic intervention and takes a liberal-ish stance with regard to social policy. The northeast quadrant indicates the exact opposite, that is a strictly Conservative type, while the southeast points to a Classical Liberal (or Libertarian) type, i.e. a person who supports extensive individual liberties and opposes state intervention in the economy. Finally, the northwest quadrant groups together those who support state economic intervention but are conservative from the social point of view. Maddox and Lielie use the term Populist to refer to this latter belief system, although we prefer to use the term Paternalist, so as to avoid the ambiguities often inherent in the previous term. 8 According to this typology, in both of the aforesaid Italian surveys almost all respondents fell between the Social Democratic and the Libertarian quadrants, while the mean position along the two dimensions clearly identifies a Social Democrat type in Italy both in 2003 (economic mean: 8.69 s.e.:.56; social mean: 6.53 s.e.:.26) and in 2006 (economic mean: 8.89 s.e.:.62; Figure 3. The mean expert position in the second scenario (Italy 2003 and 2006). 8

PPQ341051 1/9/09 11:48 pm Page 9 CURINI: EXPERTS POLITICAL PREFERENCES social mean: 7.09 s.e.:.30). Furthermore, a stable 67 percent of all respondents fell within the Social Democratic quadrant in both surveys, while roughly one-quarter of all experts fell within the Libertarian quadrant. As in the previous case, the second scenario also witnesses a fairly stable mean ideal point for both surveys. A Cross-national Comparison We repeated the unfolding analysis employed in the Italian case for every country surveyed in BL satisfying the following two conditions: 1) at least 20 people replied to the sympathy question; 2) at least 5 parties were evaluated by the experts for each country examined. The first condition guarantees a reasonable pool of country experts, 9 while the second condition recognizes the risk of generating potentially meaningless results from an unfolding analysis with a limited number of stimuli. These two conditions leave us with 21 countries out of a total of 47 surveyed (plus the 2006 Italian survey), resulting in 958 expert responses (61.6% of the total expert responses reported in BL). Table 2 reports the mean location of experts ideal points in each country analyzed under the first scenario (i.e. taking the left right dimension into account) with the corresponding standard error. We even report the overall Table 2. Cross-national comparison of the ideal points of the respondents along the Left Right dimension: 1 = left, 20 = right. Mean expert position reported Country mean s.e. obs. Italy 2003 6.02 0.40 51 Italy 2006 6.42 0.51 40 Netherlands 6.68 0.34 20 Israel 6.91 0.80 28 New Zealand 6.92 0.79 19 Switzerland 7.48 0.54 47 Belgium 7.71 0.51 21 Norway 8.20 0.70 18 Spain 8.64 0.28 75 Canada 8.65 0.30 99 Denmark 8.67 0.86 23 Finland 9.02 0.48 32 UK 9.30 0.33 58 Portugal 9.53 1.04 20 Sweden 9.99 0.46 61 Germany 10.13 0.14 91 Hungary 10.33 0.35 37 Japan 10.73 0.72 44 Overall mean 8.66 0.12 784 9

PPQ341051 1/9/09 11:48 pm Page 10 PARTY POLITICS mean for the entire sample (8.66 s.e.:.12). 10 As we can see, the mean position of Italian respondents (both in 2003 and 2006) was the furthest to the left, followed by that of the Netherlands, Israel and New Zealand, while the mean position in the case of Japan, Hungary and Germany was closest to the midpoint of the scale. The results under the second scenario are plotted in Figure 4. 11 With the partial exception of Japan, all expert means fall into one of the two lower quadrants. The coordinates of the overall mean are, respectively, 10.31 along the Economic dimension (s.e.:.15), and 6.57 along the Social Liberalism dimension (s.e.:.12). In this sense, the overall mean is situated right between the Social Democratic and the Libertarian quadrants. The highest percentage of Social Democrats can be found in the Netherlands and in New Zealand (around 90%), while the highest percentage of Libertarians can be found in Germany, Sweden and Hungary (above 70%). 12 Finally, of the patterns that emerge from Figure 4, one of the most intriguing is the rather homogeneous cluster formed by the post-communist countries included in the analysis, firmly positioned in the Libertarian quadrant. However, our main aim here is not to underline (possible) national differences between respondents. On the contrary, we want to use the experts unfolded policy preferences to investigate the possible ideological bias in their responses. Figure 4. The mean expert position in the second scenario (all countries considered). 10

PPQ341051 1/9/09 11:48 pm Page 11 CURINI: EXPERTS POLITICAL PREFERENCES Experts Preferences and the Risk of Ideological Bias: An Analysis Using the Left Right Scale As we pointed out in the introduction, expert surveys have nowadays become one of the main methods (if not the main method) of identifying the policy preferences of political parties. The central question, however, is the degree of validity of the information resulting from such surveys. The validity question was raised from a theoretical point of view by Ian Budge (2000). 13 One important concern is that the criteria on which experts base their judgements on may vary significantly. One expert may be judging the position of the electorate, while another evaluates the party leadership yet another expert may well be evaluating the views of party activists. Similarly, one expert may, for example, conceive of left right ideology in terms of a party s position on economic issues, while another may focus on social issues. There is, however, a more fundamental problem that comes before the others from the logical point of view. Indeed, an array of empirical studies argue that the estimation of party positions on the basis of survey data are not always consistent, as respondents tend to place the parties they like closer to where they perceive themselves to be, and to place those parties they dislike farther away than their actual position would warrant thus producing an ideological bias known as rationalization or projection (Granberg and Brown, 1992). In particular, assimilation effects refer to shortening the perceived ideological distance between oneself and those parties one favours, while contrast effects refer to exaggerating the distance between oneself and the parties one does not support. We have no reason to believe, a priori, that this phenomenon does not arise in an expert survey. Indeed, if the subjective political views of the respondents to an expert survey systematically interfere with their supposedly objective expert knowledge, then the very meaning of an expert survey that is, experts answers providing indications of the (unobservable) true spatial location of a party s policy position will be undermined. We need to clarify an important point here. The aim of an expert survey is not to make inferences about the attributes of the population of experts from the set of actual respondents to an expert survey, but to establish the accurate location of each party policy position using the said experts knowledge. In this sense, if the respondents to an expert survey are systematically unrepresentative of all possible experts (e.g. more left-wing or liberal than the latter), this will clearly produce a biased sample. Nonetheless, it is perfectly reasonable to have a highly unrepresentative sample (from the ideological point of view) that does not show any sign of ideological bias, or vice versa to have an ideologically representative sample characterized by a high degree of ideological bias. The former would produce useful estimates of parties positions, while the latter would not. In other words, the two problems (sampling bias and ideological bias) do not go hand in hand 11

PPQ341051 1/9/09 11:48 pm Page 12 PARTY POLITICS in the case of an expert survey. On the contrary, their relationship is a rather complex one, and thus one which we need to bear in mind. We will focus, for now, on the general left right dimension, which is generally perceived to be the main dimension of political competition. BL (pp. 164 8) propose the following method of assessing whether the scale position of a party may be systematically predicted from respondents ideological distance from that position. The authors regress experts placements on their sympathy scores for each party. They then compute the mean and the variance of the predicted left right score for an expert with an indifferent sympathy score of 10.5 (the midpoint of the 1-to-20 scale), taking this to be the corrected placement. Finally, they compare the corrected expert placements with the actual placements (i.e. the mean score of parties resulting from the expert survey) through a formal t-test to see whether or not these differences are statistically distinguishable. 14 BL only conclude that there is evidence of ideological bias if the said differences are in fact statistically distinguishable. 15 By replicating their analysis, we discovered a total of 22 biased parties (13.9% of the total, representing a higher degree of bias than was reported in BL). Table 3 summarizes these findings. By comparing the actual placement with the corrected placement, we can see that unsympathetic experts systematically place biased parties further to the right than their corrected scores would suggest. There are only two exceptions to this, and both of them, significantly, involve a left-wing party (Groen Links in Holland and Democratici di Sinistra in Italy). Note also that the extreme right parties represent 59 percent of the total (13 out of 22). 16 Thus one could argue that these results are partially a statistical artefact of the BL method. Indeed, as the same authors admit, such parties seem to invite biased perceptions by expert observers, so that the very notion of an unbiased expert observer [i.e. an expert with a sympathy score for such parties close to the indifferent threshold of 10.5] is indeed largely hypothetical (BL: 92). In other words, by setting the sympathy score at 10.5 for a neutral expert, when most (or all) the respondents lie outside of that range, may in fact be the equivalent of choosing a non-typical respondent, or even of extrapolating the relationship beyond any observed sympathy scores. So it should be no surprise to find that the corrected placement of various parties in Table 3 appears highly suspect. 17 The BL method is an ingenious way of detecting the presence of ideological bias. Nevertheless, it has one important limitation. Consider the following example: let us suppose that the true position of party j on the left right scale is around 10. Now suppose that there are just two expert respondents (or two subsets of respondents sharing a similar point of view). The first respondent considers himself as an extreme left-wing person (with, say, a score of 3 on the usual 1 20 left right scale). The second respondent is an extreme right-wing person (e.g. scoring 17 on the left right scale). Finally, let us assume that both experts are unsympathetic to party j. If both 12

PPQ341051 1/9/09 11:48 pm Page 13 CURINI: EXPERTS POLITICAL PREFERENCES Table 3. Parties for whom the difference between actual left right placement and corrected left right placement is significantly different from zero, using sympathy scores Actual Corrected S.e. of Country Party placement placement Difference difference Obs. Belgium National Front (Fn) 19.00 14.15 4.85 0.38 19 Britain Conservative Party (Con) 16.39 15.26 1.12 0.39 57 Canada Canadian Alliance (Ca) 17.46 16.36 1.10 0.52 98 Finland Perussuomalaiset (Pp) 15.35 9.90 5.45 1.50 31 Germany Republicans (Rep) 18.88 14.32 4.56 0.91 82 Germany Partei Rechtsstaatlicher Off. (Schill) 17.56 15.11 2.45 0.57 84 Germany German People s Union (Dvu) 19.42 16.27 3.15 1.45 86 Holland Christenunie (Cu) 12.35 9.59 2.76 1.12 20 Holland Lijst Pim Fortuyn (Lpf) 17.70 14.92 2.78 1.22 20 Holland Groen Links (Gl) 4.90 7.13 2.23 0.66 20 Hungary Independent Smallholders Party (Fkgp) 17.12 15.52 1.60 0.51 25 Italy 2003 Forza Italia (Fi) 15.61 13.56 2.05 0.55 51 Italy 2003 Lega Nord (Ln) 17.04 13.89 3.15 0.85 49 Italy 2003 Alleanza Nazionale (An) 16.96 15.69 1.27 0.39 51 Italy 2003 Democratici di sinistra (Ds) 5.96 6.85 0.89 0.37 50 Italy 2006 Mov. Sociale Fiamma Tricolore (Msft) 19.19 12.04 7.15 1.34 37 Italy 2006 Alternativa Sociale (As) 19.08 12.90 6.18 2.57 37 New Zealand Act New Zealand (Act) 18.16 16.20 1.96 0.60 19 Spain Partido Popular (Pp) 16.99 15.17 1.81 0.43 76 Spain Partido Nacionalista Vasco (Pnv) 14.49 13.12 1.37 0.48 75 Switzerland Schweizerische Volkspartei (Svp) 17.91 17.03 0.89 0.33 47 Switzerland Schweizer Demokraten (Sd) 18.95 15.80 3.15 1.33 42 13

PPQ341051 1/9/09 11:48 pm Page 14 PARTY POLITICS experts present signs of ideological bias as defined earlier, then the extreme left-wing respondent would probably assign a score of more than 10 to party j, while the extreme right-wing respondent would assign it a lower score. In this case, however, regressing their sympathy scores on the left right scores (as was done following the BL method) would produce, on average, a zero effect. In other words, by focusing on the sympathy question alone, we run the risk of underestimating (or overestimating, depending on the case in question) the true degree of ideological bias. By using the results of the unfolding analysis instead, and thus by focusing directly on the relationship between the respondents ideal points and the parties ideological stance, we can avoid the aforementioned risk. Besides, by building on this we can even provide a method of correcting the scores for the biased parties. In this regard, the extent of ideological bias may be assessed by decomposing the variance of party placement over respondents. If Policy i denotes the respondent s position on the left right scale (as results from the previous unfolding analysis) and Score i denotes the mean party position on the same scale, then the effect of ideological bias could be expected to be proportional to (Policy i Score i ) for a sympathetic party, and the negative value of this quantity for an unsympathetic party. (See Markus and Converse, 1979, and Merrill and Grofman, 1997, for similar analyses using mass surveys). More specifically, we fitted the following model for each party from those countries shown in Table 2: Score ij = α j + β j [s(policy i Score i )] + ε i (1) where s can assume two values: +1 if the sympathy score of respondent i for party j is less than or equal to 10; 1 if it is higher than 10. As a consequence, a β j coefficient that is statistically distinguishable from zero may be considered an indication of ideological bias towards that particular party among the respondents. Our results would indicate that this is far from unusual. On the contrary, the estimated coefficients are indeed significant (at the usual 95% confidence level) for 53 parties out of 158 (33.5% of the total). In two-thirds of such cases, the parties involved are right-wing parties (i.e. with a left right score greater than 10). On average, our model explains 7 percent of the variance in party placement. However, the said explained variance increases to almost 20 percent if we consider biased parties only. The advantage of this method is that it points to a direct way of correcting biased scores, i.e. of neutralizing the effects of ideological bias. This can be achieved by subtracting the estimate of the respondent s ideological bias (i.e. β j ) from each respondent s placement of any given party (i.e. from Score ij ). By following this procedure for each of the parties covered by our analysis, we can compute an ( unbiased ) mean for each party (along with its standard error). Among other things, this enables us to establish whether the detected ideological bias has any relevant (and not merely significant) impact on the parties positions. 14

PPQ341051 1/9/09 11:48 pm Page 15 CURINI: EXPERTS POLITICAL PREFERENCES This difference is of substantial analytical importance. Indeed, notwithstanding the evidence of ideological bias, the mean location of parties on a given issue scale can still provide a good estimate of all respondents perceptions of a party s stand, if the rationalization effects that push and pull individual party placement (i.e. the assimilation and contrast effects involved in the ideological bias) are not overwhelmingly in one direction only. On the other hand, if this did not happen, then the push and pull would not cancel each other out across respondents, as a result of which any (aggregate) measure would be far less robust. As in Table 3, Table 4 also shows those parties whose actual mean (as results from the survey) is, at a 95 percent confidence interval, different from our unbiased mean. As the table shows, there are 25 such cases out of a total of 158 (i.e. approximately 1 case for every 6.5 parties). Once again, the extreme right-wing parties constitute a significant proportion of the entire set (12 out of 25). However, unlike in Table 3, the (unbiased) placements, even for those parties, now seem far more reasonable. 18 With regard to the specific nature of ideological bias, Table 4 reveals that contrast effects always seem to dominate attraction effects. Indeed, a comparison of the actual mean with the unbiased one shows that all parties are placed at a more extreme value than their unbiased scores would suggest (the only exception being the far-right German National Democratic Party). Finally, the overlap between the set of biased parties drawn up on the basis of the BL method (Table 3) and that drawn up using our method (Table 4) is not perfect (involving only 14 parties out of 25); this would seem to suggest that by neglecting the (underlying) policy preferences of experts, misleading results may be produced. We examined these findings in greater depth by estimating a rare events (Re)Logit. 19 In particular, we modelled the probability of party j being biased (as results from Table 4) as a function of: its (actual) left right score (LEFT RIGHT); of the distance (in absolute terms) between its left right score and the average placement of experts in party j s country (as reported in Table 2) (DISTANCE); and, finally, as a function of the variance of the experts ideal points in party j s country (VARIANCE). This last variable, in particular, underlines the importance of relying on a range of different respondents points of view if the risk of ideological bias is to be reduced. The results are shown in Table 5. DISTANCE and VARIANCE are both highly significant, and both coefficients present the expected sign: as DISTANCE increases, or as VARIANCE decreases, the probability of party j being biased increases. On the other hand, the ideological stance of party j is not significant. 20 For example, according to our model, the marginal effect of increasing DISTANCE by one standard deviation (with the other variables held at their mean) is about +18% (the equivalent of a 26.1% probability of being biased), whereas modifying the left right position of a party has no effect distinguishable from zero. In other words, it is not the left right position of a party per se that affects its chances of being biased. What really matters 15

PPQ341051 1/9/09 11:48 pm Page 16 PARTY POLITICS Table 4. Parties for which the difference between actual left right placement and Unbiased left right placement is significantly different from zero, using experts ideal points Actual Unbiased S.e. of the Country Party placement placement Difference difference Obs. Belgium National Front (Fn) 19.00 17.04 1.96 0.41 19 Belgium Flemish Block (Vb) 18.85 17.34 1.51 0.64 20 Britain Conservative Party (Con) 16.39 15.33 1.05 0.33 56 Canada Canadian Alliance (Ca) 17.46 16.35 1.11 0.26 98 Canada Progressive Conservative Party of Canada (Pc) 13.64 13.05 0.60 0.26 98 Germany Partei Rechtsstaatlicher Off. (Schill) 17.56 16.21 1.35 0.24 80 Germany National Democratic Party (Npd) 19.70 19.91 0.21 0.07 78 Holland Volkspartij Voor Vrijheid en Democratie (Vvd) 16.40 15.09 1.31 0.51 20 Holland Christen Democratisch Appe l (Cda) 13.60 12.58 1.02 0.44 20 Holland Lijst Pim Fortuyn (Lpf) 17.70 16.29 1.41 0.64 20 Hungary Independent Smallholders Party (Fkgp) 17.12 15.90 1.22 0.40 25 Hungary Workers Party (Munkàs) 2.17 3.40 1.22 0.49 29 Israel United Arab List (Raam) 3.23 5.12 1.89 0.69 22 Italy 2003 Lega Nord (Ln) 17.04 14.61 2.43 0.36 49 Italy 2003 Alleanza Nazionale (An) 16.96 15.47 1.49 0.29 51 Italy 2003 Forza Italia (Fi) 15.61 13.95 1.66 0.43 51 Italy 2006 Alternativa Sociale (As) 19.08 18.32 0.76 0.22 37 Italy 2006 Forza Italia (Fi) 14.98 14.21 0.76 0.29 40 Italy 2006 Nuovo Partito Socialista Italiano (Npsi) 11.78 10.49 1.29 0.55 32 Italy 2006 Lega Nord (Ln) 17.30 16.48 0.82 0.38 40 New Zealand Act New Zealand (Act) 18.16 16.09 2.06 0.46 19 New Zealand New Zealand National Party (Np) 14.58 12.90 1.68 0.48 19 Spain Partido Popular (Pp) 16.99 14.63 2.35 0.31 75 Spain Partido Nacionalista Vasco (Pnv) 14.49 13.48 1.01 0.40 74 Switzerland Schweizer Demokraten (Sd) 18.95 18.27 0.68 0.31 42 16

PPQ341051 1/9/09 11:48 pm Page 17 CURINI: EXPERTS POLITICAL PREFERENCES Table 5. (Re)Logit estimates for parties probabilities of being biased (s.e. clustered over countries) Model estimates Coefficient s.e. Intercept 3.69**.879 Left-right.030.07 Distance.438**.105 Variance.137*.045 n = 158 ** p < 0.01. * p < 0.05. is the relationship between that position and the specific attributes of the set of respondents (i.e. the range of their policy preferences and the distance of the party from the average respondent s ideal point). 21 From the Left Right Scale to More Specific Policy Dimensions In the previous section we showed that the ideological bias in the BL expert survey is not negligible, as it involves almost 16 percent of the parties in question. This bias is relevant because it can jeopardize the usefulness of expert scores for the purposes of empirical analysis, given that we no longer have a valid position for the biased party. For example, if the presence of bias implies a different ranking of parties along the left right scale, the location of the median party within a given party system may prove problematic, and so on. We have even proposed a method designed to help correct the biased scores based on the unfolded experts ideal points. There is, of course, another layer of analysis (and hence uncertainty) involved in using the respondents indirectly estimated ideal points as opposed to their directly observed sympathy scores. In this sense, our approach can be seen as a valid solution to the ideological bias in the data, as long as the estimated ideal points suitably (and precisely) represent the true ideal points of the respondents. We have already seen how the fit measures of the unfolding analysis reassure us on this point. Furthermore, in the absence of any direct evidence of the respondents ideal points, the procedure we followed, although imperfect, seems to be the most promising method of generating less biased estimates. However, there is still a second option available. Up until now we have focused exclusively on the left right scale. However, if we now move from the left right scale to less abstract policy dimensions, the respondents ideological bias appears far less pronounced. For example, in the scenario envisaged in Figure 4, only 3 parties out of 158 (less than 2%) appear, according to our model, to be biased with regard to both policy dimensions reported therein (i.e. the Economic and Social dimensions). 22 If we apply the original 17

PPQ341051 1/9/09 11:48 pm Page 18 PARTY POLITICS BL approach to an analysis of the same scenario, the total number of biased parties decreases even further (leaving us with just one biased party in both dimensions, namely the Belgian National Front). 23 This difference (in terms of ideological bias) is not simply the consequence of the fact that, unlike substantially defined policy scales, the concept of left right has no secure anchor whereby respondents may determine a homogeneous, intersubjective frame of reference (McDonald, Mendes and Kim, 2007; see also Huber and Inglehart, 1995). This fact alone would eventually result merely in an increased cross-expert (and, perhaps, crossnational) variation in parties left right scores compared to other policy dimensions, not in an amplified ideological bias. 24 In this sense, there seems to be something peculiar inherent in the left right scale, but not in the other policy scales, that results in the (political and ideological?) preferences of (certain) respondents interfering more easily with their expert knowledge. As a consequence of this finding, one alternative solution to the problem of obtaining unbiased left right scores would be to construct a left right scale from the scores of political parties along those (policy) dimensions with a more precise substantive meaning and which appear less likely to generate biased answers among respondents, 25 as we have just pointed out. This could be achieved by starting with an a priori definition of the policy content of the left right scale (a combination of economic, social and foreign policy, for example), as has been recently tried, for example, by the Manifesto Research Group (Budge et al., 2001). 26 Alternatively, we could follow the kind of Vanilla approach originally proposed by Gabel and Huber (2000), and use this to inductively define the left right scale as being the one (latent) dimension that maximizes the explained variance of a set of substantive policy dimensions by certain data-reduction processes. Both alternatives clearly have their own particular weaknesses (for example, which a priori definition of the policy content of the left right scale should be considered the most appropriate? Should it be the same for each country and for each time-scale? See the discussion in Budge et al., 2001). 27 Nevertheless, they still represent valuable alternatives for third-party users of expert surveys, and as such they merit serious consideration, especially in those cases where respondents ideological bias seems to be more pronounced. 28 Finally, the present analysis seems to suggest that future designers of expert surveys would be advised to include in their surveys one question designed to uncover (either indirectly through the inclusion of a sympathy question, or in a more direct way) the respondents ideal points along certain significant policy dimensions. This would make it possible, as we have shown, to develop a further check on the validity of the estimates resulting from expert surveys. 18

PPQ341051 1/9/09 11:48 pm Page 19 CURINI: EXPERTS POLITICAL PREFERENCES Conclusion Expert surveys have become increasingly popular among political scientists for a variety of good reasons. However, even this method, designed to identify the policy preferences of parties, is not without its flaws. In this paper, we have used the latest expert survey by Benoit and Laver (2006) to unfold the ideal points of the said survey s respondents. The estimated ideal points have been employed to show that in almost 16 percent of the cases in question, there is evidence of ideological bias in the experts placement of parties along the left right dimension, especially among right-wing parties (albeit not necessarily extreme right-wing parties). Moreover, the left right position of a party appears to be less important than the particular pattern of respondents policy preferences when it comes to explaining the presence of the said bias. We have also examined two ways of producing less biased estimates. The first method is directly based on a regression technique, while the second is based on the negligibility of ideological bias in the experts answers to more specific policy questions (such as those regarding Economic or Social policies). Thus, while our analysis would seem to suggest that any precise assessment of the positions of parties along the left right scale should be treated with caution, especially in certain given cases, assessment is nevertheless reassuring with regard to the use (and the usefulness) of experts party scores along other, more specific, policy dimensions. Notes I thank Kenneth Benoit and two anonymous referees for helpful comments and discussions. I also gratefully acknowledge the financial assistance of the Italian Ministry for Research and Higher Education, Prin 2005 prot. 2005141220. All data to replicate this study, as well as tables omitted because of space limitations, are available at: http://www.sociol.unimi.it/docenti/curini/experts. 1 We would like to thank Kenneth Benoit and Michael Laver for having kindly provided us with the data. 2 The analysis was conducted using the preference scaling algorithm (PREFSCAL) included in SPSS. This algorithm has the advantage that it fits the ideal point model for unfolding, thus avoiding trivial, degenerate solutions (Busing et al., 2005). 3 Warwick (2002) terms this second dimension social control. The essence of this dimension is whether society should impose norms or rules to control individual behavior or, alternatively, accommodate as much personal freedom and individualism as possible. This dimension is sometimes even referred to as a libertarian/ authoritarianism or social authoritarianism dimension. 4 A similar two-dimensional space for political competition is developed, among others, in Keman (2007) and Marks et al. (2006). 5 The improvement we obtain in terms of fit measures using an optimal (but theoretically unintelligible) solution, compared with our constrained (but theoretically 19

PPQ341051 1/9/09 11:48 pm Page 20 PARTY POLITICS intelligible) unfolding analysis, are minimal when a single dimension is used, although they increase slightly if we take two dimensions. 6 The Pearson correlation between the disparities and the scaled distance is.83 (Italy in 2003) and.81 (Italy in 2006) in the first scenario and.83 and.77 in the second scenario, while the Spearman rank is, respectively,.82 and.79 (first scenario) and.82 and.77 (second scenario). Both correlations assess the degree to which the scaled distances (from the unfolding analysis) are monotonic to the dissimilarities data: a higher value implies a better goodness-of-fit. With regard to the badness-of-fit measure, the penalized-stress value (Busing et al., 2005) is slightly smaller in the second scenario (.66 and.71 respectively) than in the first one (.73 and.74). Smaller values indicate better scaling solutions. 7 The two-sample Kolmogorov-Smirnov test of the equality of distribution functions confirms this (p-value:.959). We cannot therefore reject the null hypothesis whereby the two distribution functions are the same. 8 See Riker (1982). 9 The only two exceptions are Norway (18 respondents) and New Zealand (19 respondents). In both cases, however, the response rate is higher than the mean response rate for the entire sample. On the other hand, we did not consider Turkey, despite its having 29 respondents, because of its low rate of response (9%). 10 As the fit of the unfold analysis improves, our confidence in its results (including the location of the mean) increases, as does confidence in their comparability. Thus, Table 3 does not show those countries displaying a low measure of fit (i.e. a Pearson correlation and a Spearman rank of less than.65). The said countries are Ireland, Poland, Slovenia and the Czech Republic. 11 The economic dimension in Figure 4 comes from one of two sources, depending on the type of country. For most countries, economic policy is represented by the tax cuts versus spending increases dimension (as in the Italian case). For postcommunist countries, however, we employed the state ownership of business and industry versus privatization dimension following BL. In order to represent liberal versus conservative social policies, we used the aforementioned social dimension, with the exception of New Zealand where social policy was not measured. In this latter case, following BL once again, we employed the immigration dimension. 12 Under the second scenario, the average Pearson correlation is.81, the average Spearman rank is.79, and the average penalized stress value is.65. Under the first scenario, these values are, respectively,.74,.72 and.74. In New Zealand, Sweden, the Czech Republic and, most notably, Ireland and Poland, the second scenario improves results considerably when compared to the first. Vice versa, in Japan and Denmark the goodness-of-fit of the model is better under the first scenario. 13 See also Mair (2001) and Steenbergen and Marks (2007). 14 Laver and Hunt (1992), in the first study of this sort, simply regress each expert s assessment of the scale position of a party against his/her self-assessed ideological distance from that party (i.e. the sympathy score ). 15 For each party, we included in the analysis just the respondents who were able to locate the party along the left right scale, and who also gave a sympathy score for that party. 16 The labelling of a party as an extreme-right party is, of course, quite arbitrary. In this paper we have defined such a party as one with a mean score of more than 17 on the left right scale. 20