Review of the policy utility of the Worldwide Governance Indicators for the Central American Countries 1

Transcription

1 Review of the policy utility of the Worldwide Governance Indicators for the Central American Countries 1 June 11, 2008 Working Paper This note was prepared by Christiane Arndt at the Maastricht Graduate School of Governance and Carmen L. Romero in LCSPS and, with comments from Nick Manning, James Anderson, Steve Knack, Charles Oman and Jeffrey Rinne and very useful suggestions from Felipe Saez, Linn Hammergren, Deborah L. Wetzel, Chris Parel, Ana Bellver, David F. Varela, Jose Eduardo Gutierrez Ossio and Omowunmi Ladipo. The authors would like to thank Dani Kaufmann and Aart Kraay for their frank and useful feedback on an earlier draft. 1

2 Table of Contents Summary... 3 I. Background... 4 II. Introduction... 6 III. Governance Indicators Potential purposes... 8 Purposes 1 and 2: Comparability across countries and over time... 8 Purpose 3: Identify reforms Purpose 4: Research Purpose 5: World Bank credibility and country dialogue IV. So where does that leave us? Conclusion and recommendations Alternatives and complements Working with what we have Moving ahead creating a more robust set of indicators Conceptual challenges and the debate about actionworthiness V. References: Annexes Differences between the Central American countries How the Central American countries compare to other Latin American countries.. 54 Changes over-time in Central American countries Sources 2006 by Worldwide Governance Indicator Weights 2006 by Source and Worldwide Governance Indicator

3 SUMMARY Governance indicators have become more prominent in discussion about the design and implementation of public policies in developing and middle income countries over the past ten years. In response to the increased demand for more and better governance indicators a wide variety of indicators has been developed. The most commonly known are the Worldwide Governance Indicators (WGIs). Yet, the inclusion of these indicators in development analysis has raised several questions over the degree to which they reflect the real situation of governance in a country, and the inputs they provide to developmental approaches. This note examines the degree to which the WGIs satisfy five potential purposes of governance indicators in Central America. It reaches some nuanced conclusions: 1. Do the WGIs identify whether countries share similar governance characteristics and qualities and therefore assist in drawing lessons for peer learning? They do not identify whether countries have similar governance systems. They do give an indication of how countries compare to each other with respect to the quality of their governance - within certain limitations due to differences in sources across countries and conceptual problems. 2. Do they provide a time series to identify whether countries are improving? Over-time changes in the WGIs can indicate an actual changes in the quality of governance, but can also be due to changes in the composition of underlying sources, changes in underlying sources criteria and methodology, changes in other countries ratings, and changes in the perception of governance, independent of actual changes in governance. A change in the WGIs should therefore be the starting point of a deeper analysis, not the endpoint. 3. Are they sufficiently disaggregated to assist in identifying which reforms are most needed in a given country to improve its quality of governance? Generally not. 4. Are they useful for upstream research purposes? They can indeed challenge or support common perceptions, but do not allow for identifying sequencing issues in specific reforms and relationships between improvements in specific governance areas 5. Do they reinforce the Bank's credibility as a source of expertise and so enhance country level dialogue? They can, if they are carefully presented as a first cut of experts perceptions of the quality of governance and not as the final word. The note goes on to give recommendations for the short and medium-to long run. It reviews other indicators and the current trend towards the development of "disaggregated, actionable indicators" that would meet more of these purposes, but remains agnostic about whether the lack of progress in that endeavor to date is just because of its sheer scope, lack of resources, coordination and effort, or whether it stems from a fundamental conceptual problem. 3

4 I. BACKGROUND 1. Conditions for good governance in Central America have improved considerably over the last ten years. New governments have prioritized good governance and reinforced the reform processes. However, there are particular governance concerns for Guatemala where the consolidation of democracy is still fragile and problems related to the impoverished indigenous population remain largely unsolved. Nicaragua and Honduras also continue to face important governance challenges. 2. The WGI data shows little advance in the quality of governance for the Central American countries (see annex 1). The average percentile rank for the Central American countries (Costa Rica, El Salvador, Guatemala, Honduras, Nicaragua and Panama) continues to be below the Latin American regional average (See figures 1 and 2). Figure 1 Figure 2 Indicator by LC2 Country Average by Indicator and by Region Cost of Corruption Rule of Law Regulatory Quality Government Effectiveness Political Stability Voice and Accountability Cost of Corruption Rule of Law Regulatory Quality Government Effectiveness Political Stability Voice and Accountability COSTA RICA EL SALVADOR GUATEMALA HONDURAS NICARAGUA PA NAMA World LAC LC2 3. There are large differences in the performance of the Central American countries on the WGIs: while many Central American countries score poorly in particular on the Control of Corruption, Rule of Law and Government effectiveness WGIs, Panama and Costa Rica are above the 50th percentile for almost all indicators. Costa Rica stands out for achieving the highest percentile rank in the Latin American region for Political Stability, while Guatemala is ranked below the 15th percentile for Rule of Law. 4

5 Figure 3 Voice and accountability Political stability Government effectiveness Regulatory quality Rule of law Control of corruption Percentile Rank (0-100) Latin America and Caribean World Costa Rica Panama El Salvador Guatemala Honduras Nicaragua 5

6 II. INTRODUCTION 4. The Country Management Unit for Central America commissioned this informal note to examine the utility of the Worldwide Governance Indicators in the dialogue with countries in that region. 5. Sidestepping the massive problems inherent in defining governance, broadly (very broadly) the task of the Bank in relation to the public sector is to work with the executive, the judiciary and, with caution, the legislature seizing opportunities to improve efficiency and effectiveness in the use of public expenditures to reduce poverty and improve economic and social opportunity. Work to reduce corruption is an intrinsic part of that task. 2 In order to assess the usefulness of any indicators, it is necessary to start by speculating on how any indicators of governance could, in principle, assist the Bank in these areas of policy dialogue with clients. 6. There are perhaps five non-exclusive contributions that indicators of institutional strength and/or performance could make: 1. Cross-country comparable indicators could identify whether countries have similar governance characteristics and qualities and therefore assist countries in learning from the success or failure of their peers. 2. Time series indicators could identify whether countries are improving; 3. Disaggregated or disaggregatable country-level indicators could identify which reforms are most needed in a given country; 4. Any combination of the above could, in principle, be used for upstream research purposes to challenge or support common perceptions, to identify sequencing issues in reform (e.g., do public expenditure reforms precede civil service reforms, etc.?) and apparent relationships (e.g., the relationship between improvements in public expenditure management and improved service delivery, etc.) 5. Any combination of the above can be used to reinforce the Bank's credibility as a source of expertise and so enhance country-level dialogue. 7. The recent World Bank Governance and Anti-corruption report emphasizes some of these purposes (World Bank 2007): It notes that the "Development Committee asked the Bank to further develop and use disaggregated and actionable indicators." In promoting this, the GAC report is emphasizing purposes 1 and 3, as it emphasizes the need for indicators to "inform the 2 To note some key terms: Governance concerns how the state obtains and uses authority in providing goods and services, in creating the conditions for economic and social development, in maintaining social order and providing justice, and in maximizing individual liberties consistent with those other goals. Accountability to others (constituents, managers, elected politicians, lawmakers) is a common institutional arrangement and corruption is a symptom of failure of such accountability arrangements. Transparency is a common device for enhancing accountability. 6

7 CPIA" (i.e., determine which reforms are most needed in a given country) and "help track progress in specific reforms implemented by governments" (i.e., time series). However, it also refers to "aggregate worldwide governance indicators (gathered by DEC and WBI)" but sees these as "research products that draw on a wealth of information from non-wbg (World Bank Group) efforts to monitor governance," suggesting that these are valuable for purposes 4 and 5. It takes a somewhat cautious line concerning DEC and WBI indicators when it notes that they "exist alongside many other products (e.g., enterprise surveys) to provide comparative information for policymakers; they are not official ratings by the WBG, and they have no formal role in any WBG operational decision-making." 8. This paper looks at the part that the WGIs play in meeting these 5 potential purposes of indicators in Central America and at possible alternatives. It builds on Arndt and Oman (2006). See Pollitt (forthcoming) for another review of the usefulness of the WGIs with respect to different purposes and Knack (2007) for a deep review of the WGIs and other indicators for Eastern Europe and Central Asia. Lessons drawn are also relevant for other regions. 7

8 III. GOVERNANCE INDICATORS POTENTIAL PURPOSES Purposes 1 and 2: Comparability across countries and over time 9. Do the WGI indicators identify whether countries share similar governance characteristics and qualities and therefore assist in drawing lessons for peer learning? Do they provide a time series to identify whether countries are improving? 10. The answer is that the WGIs provide limited answers to these questions. This section first explains the findings of the WGIs for Central American countries. It then explains in a second step why any findings of the WGIs with respect to comparability over time and across countries should be carefully examined.: i) the criteria for rating countries on the WGIs differ across countries and over time and ii) there are several potential reasons for changes in country ratings on the WGIs over time that are independent of actual changes in the quality of governance. This section also attempts to iii) raise awareness of the normative content of the WGIs that users automatically endorse in drawing conclusions about cross-country differences and over-time changes in the quality of governance on the basis of the WGIs. 11. It concludes that while significant cross-country differences and changes over time on the WGIs can indicate actual differences and changes over time in the quality of governance, a deeper analysis of the reasons for these changes and differences is necessary before drawing any conclusions about a country s quality of governance. Kaufmann et al. frequently emphasize in their publications the importance of looking carefully at the underlying data sources and to complementing their indicators with more in-depth and country-specific analysis. Findings of the WGIs for Central American countries 12. This section looks at findings of i) differences in the quality of governance between Central American countries, ii) the performance of the Central American countries with respect to their income category average, iii) changes over time in the quality of governance in Central American countries and iv) Central America s standing with respect to other Latin American countries. 13. Box 1 explains how the WGIs are constructed. Kaufmann et al. warn users that crosscountry differences and over-time changes are only significant if the confidence intervals, based on estimates of the measurement error (see Box 1), do not overlap. Of course measurement error is not unique to the WGIs. We will apply their rule throughout this section and discuss it in the next section. 8

9 Box 1 The World Bank Institute s Worldwide Governance Indicators The six Worldwide Governance Indicators (WGIs), namely Voice and Accountability, Political Stability, Government Effectiveness, Regulatory Quality, Rule of Law and Control of Corruption, are the most popular governance indicators. Publicly available and easily accessible on the web, they are widely-used to compare the quality of governance over time and across countries, for aid-allocation decisions, for risk ratings, for academic analysis and for media articles. The authors, Dani Kaufmann, Aart Kraay and Massimo Mastruzzi, summarize data from more than 30 expert assessments and household and firm surveys. The WGIs are available from 1996 to today for more than 200 countries. Figures 4a and 4b (see below) illustrate the way a WGI is constructed, using the example of the ratings on Regulatory Quality for Guatemala, Honduras, Nicaragua and Panama for 1996 and 2006 (Costa Rica has a different set of sources for 1996 and El Salvador for 2006). Tip: Rating Regulatory Quality At the tip of the pyramid is the rating for Regulatory Quality. This indicator ranges roughly from -2.5 to +2.5 (the higher the better the Regulatory Quality in a country) with no specific meaning of the scale, i.e. we cannot derive from a country s rating of -1 that the country is characterized by a set of Regulations A and that a country with a rating of +1 is characterized by a set of Regulations B. 1st layer: Sources Underneath the overall rating are the primary sources of information - surveys or expert assessments - from which that rating is calculated. The rating for Guatemala, Honduras, Nicaragua and Panama is based on the ratings of 10 sources (9 expert assessments and one firm survey) in 2006 and on 5 sources (4 expert assessments and 1 firm survey) in 1996, the 1st layer. Each source s weight in the index depends on how well it tends to agree with other sources, not specifically for the four Central American countries, but for all countries with available data. The more highly correlated a source is with other sources, the higher the weight it receives. An important feature of the WGI indexes is that ratings for one year are usually based on a somewhat different set of sources to the ratings for an earlier or later year. To illustrate, only half of the sources were available in 1996 for Panama, Guatemala, Honduras and Nicaragua, compared to Similarly, since most sources are available only for a limited number of countries, different countries ratings are based on different sources. To illustrate, for the WGI indicator Rule of Law, out of the 17 sources used for Guatemala, only 10 are used to assess the Rule of Law in the United States. Weights assigned to each source also differ across countries and over time, for two reasons. First, where there is data from sources missing for a country, the weights for available sources are scaled up, and missing data differ across countries and over years for a given country. Second, correlations among sources vary somewhat from year to year, so the weights would change even for a theoretical case in which data were available for all sources and all years for a given country (see Figure 14 and 15). 9

10 2nd layer: Underlying indicators Most primary sources are themselves composite indicators, constructed from the answers of experts, business and households to survey questions or checklists, as shown in Figure 4. Each primary source therefore incorporates many underlying indicators which can be thought of as the lower layer of the pyramid. In many cases users will not have access to countries scores on these underlying indicators or to the criteria used to rate each country on each underlying indicator. Margins of Error: In addition to each country s rating, Kaufmann et al. provide margins of error to represent the reliability of the rating. They construct 90 % confidence intervals on the basis of these margins of error around a country s rating which, according to their estimate, has a 90 per cent probability of containing the country s true (as opposed to estimated) score. Kaufmann et al. warn users not to compare countries with each other or over time where the confidence intervals overlap. The margins of error and the corresponding confidence intervals decrease with i) an increase in the number of sources and ii) a higher correlation among the sources. Hence, the more sources Kaufmann et al. use and the more these sources agree on countries ratings, the more reliable Kaufmann et al. believe their indicators to be. For more details, see Kaufmann et al. (2007a, 2007b), Arndt and Oman (2006), Knack (2007) 10

11 Figure 4 a) Regulatory Quality 2006 Panama, Guatemala, Honduras, Nicaragua Upper Layer IFAD: Rural Sector Performance Assessment (IFD) Bertelsmann Foundation: Bertelsmann Transformation Index Global Insight: Global Risk Service Economist Intelligence Unit (EIU): Country Risk Service World Economic Forum (WEF): Global Competitiveness Report, Executive Opinion Survey Heritage Foundation/Wall Street Journal: Economic Freedom Index Merchant International Group: Grey Area Dynamics World Bank (WB): Country Policy and Institutional Assessments Political Risk Services (PRS): International Country Risk Guide Global Insight: Business Conditions and Risk Indicators Lower layer Price stability (Bertelsmann Foundation) Private Property (Bertelsmann Foundation) Policies Non-Tax: Regulations -- Exports: A 2% reduction in export volume as a result of a worsening in export regulations or restrictions (such as export limits) during any 12-month period, with respect to the level at the time of the assessment. (Global Insight Global Risk Service) Policies Non-Tax: Regulations -- Imports: A 2% reduction in import volume as a result of a worsening in import regulations or restrictions (such as import quotas) during any 12-month period, with respect to the level at the time of the assessment. (Global Insight Global Risk Service) Policies Non-Tax: Regulations -- Other Business: An increase in other regulatory burdens, with respect to the level at the time of the assessment, that reduces total aggregate investment in real LCU terms by 10% (Global Insight Global Risk Service) Policies Non-Tax: Ownership of Business by Non-Residents: A 1-point increase on a scale from "0" to "10" in legal restrictions on ownership of business by non-residents during any 12- month period. (Global Insight Global Risk Service) Policies Non-Tax : Ownership of Equities by Non-Residents: A 1-point increase on a scale from "0" to "10" in legal restrictions on ownership of equities by non-residents during any 12-month period. (Global Insight Global Risk Service) Tax Effectiveness: How efficient the country s tax collection system is. The rules may be clear and transparent, but whether they are enforced consistently. This factor looks at the relative effectiveness too of corporate and personal, indirect and direct taxation. (Global Insight Global Risk Service) Legislation: An assessment of whether the necessary business laws are in place, and whether there any outstanding gaps. This includes the extent to which the country's legislation is compatible with, and respected by, other countries' legal systems. (Global Insight Global Risk Service) Unfair competitive practices (EIU) Price controls (EIU) Discriminatory tariffs (EIU) Excessive protections (EIU) Discriminatory taxes (EIU) Administrative regulations are burdensome (WEF) Tax system is distortion (WEF) Import barriers as obstacle to growth (WEF) Competition in local market is limited (WEF) It is easy to start company (WEF) Anti monopoly policy is lax and ineffective (WEF) Environmental regulations hurt competitiveness (WEF) Cost of tariffs imposed on business (WEF) Government subsidies keep uncompetitive industries alive artificially (WEF) Complexity of Tax System (WEF) Domestic banks are protected from foreign competition (WEF) Barriers to entry in banking sector are very high (WEF) Interest rates are heavily regulated (WEF) Private sector participation in infrastructure projects is not permitted (WEF) Costs of uncertain rules, laws, or government policies (WEF) Transfer costs associated with exporting capital as an obstacle to business (WEF) General uncertainty on costs of regulations as an obstacle to business (WEF) Openness of public sector contracts to foreign investors (WEF) Policies for dividend remittances as obstacles to development (WEF) Dominance of state owned or state controlled enterprise (WEF) State interference in private business (WEF) Regulatory discretionally (WEF) Price controls as an obstacle to business development (WEF) Regulations on foreign trade as an obstacle to business development (WEF) Foreign currency regulations as an obstacle to business development (WEF) Easy to start a company (WEF)) Wage/ Prices (Heritage Foundation/Wall Street Journal) Foreign investment (Heritage Foundation/Wall Street Journal) Banking/Finance (Heritage Foundation/Wall Street Journal) Unfair Trade (Merchant International Group) Unfair Competition (Merchant International Group) Competitive environment (World Bank CPIA) Factor and products markets (World Bank CPIA) Trade policy (World Bank CPIA) Investment Profile: Includes the risk to operations (scored from 0 to 4, increasing in risk); taxation (scored from 0 to 3), repatriation (scored from 0 to 3); repatriation (scored from 0 to 3) and labour costs (scored from 0 to 2). They all look at the government s attitude towards investment. (PRS ICRG) Tax Effectiveness: How efficient the country s tax collection system is. The rules may be clear and transparent, but whether they are enforced consistently. This factor looks at the relative effectiveness too of corporate and personal, indirect and direct taxation. (Global Insight: Business Conditions and Risk Indicators) Legislation: An assessment of whether the necessary business laws are in place, and whether there any outstanding gaps. This includes the extent to which the country's legislation is compatible with, and respected by, other countries' legal systems. (Global Insight: Business Conditions and Risk Indicators) Enabling conditions for rural financial services development (IFD) Investment climate for rural businesses (IFD) Access to agricultural input and produce markets (IFD) 11

12 Figure 4 b) Regulatory Quality 1996 Panama, Guatemala, Honduras, Nicaragua Upper Layer Economist Intelligence Unit (EIU): Country Risk Service World Economic Forum (WEF): Global Competitiveness Report, Executive Opinion Survey Heritage Foundation/Wall Street Journal: Economic Freedom Index World Bank (WB): Country Policy and Institutional Assessments Political Risk Services (PRS): International Country Risk Guide Lower layer Unfair competitive practices (EIU) Price controls (EIU) Discriminatory tariffs (EIU) Excessive protections (EIU) Discriminatory taxes (EIU) Administrative regulations are burdensome (WEF) Tax system is distortion (WEF) Import barriers as obstacle to growth (WEF) Competition in local market is limited (WEF) It is easy to start company (WEF) Anti monopoly policy is lax and ineffective (WEF) Environmental regulations hurt competitiveness (WEF) Cost of tariffs imposed on business (WEF) Government subsidies keep uncompetitive industries alive artificially (WEF) Complexity of Tax System (WEF) Domestic banks are protected from foreign competition (WEF) Barriers to entry in banking sector are very high (WEF) Interest rates are heavily regulated (WEF) Private sector participation in infrastructure projects is not permitted (WEF) Costs of uncertain rules, laws, or government policies (WEF) Transfer costs associated with exporting capital as an obstacle to business (WEF) General uncertainty on costs of regulations as an obstacle to business (WEF) Openness of public sector contracts to foreign investors (WEF) Policies for dividend remittances as obstacles to development (WEF) Dominance of state owned or state controlled enterprise (WEF) State interference in private business (WEF) Regulatory discretionally (WEF) Price controls as an obstacle to business development (WEF) Regulations on foreign trade as an obstacle to business development (WEF) Foreign currency regulations as an obstacle to business development (WEF) Easy to start a company (WEF)) Wage/ Prices (Heritage Foundation/Wall Street Journal) Foreign investment (Heritage Foundation/Wall Street Journal) Banking/Finance (Heritage Foundation/Wall Street Journal) Unfair Trade (Merchant International Group) Unfair Competition (Merchant International Group) Competitive environment (World Bank CPIA) Factor and products markets (World Bank CPIA) Trade policy (World Bank CPIA) Investment Profile: Includes the risk to operations (scored from 0 to 4, increasing in risk); taxation (scored from 0 to 3), repatriation (scored from 0 to 3); repatriation (scored from 0 to 3) and labour costs (scored from 0 to 2). They all look at the government s attitude towards investment. (PRS ICRG) 12

13 Differences between the Central American countries 14. The WGIs are not informative about governance systems, but are outcome measurements of the quality of governance. In fact, countries having similar scores can have very different governance arrangements (Andrews, 2008). Figure Costa Rica and Panama score significantly better on Voice and Accountability than El Salvador, Nicaragua, Guatemala and Honduras, according to Kaufmann et al. (2007a). Differences among the latter are not statistically significant in terms of percentile rank, as shown in Figure 5 which graphs the percentile rank of the Central American countries according to the Voice and Accountability indicator. Percentile rank indicates the percentage of countries worldwide that rate below the selected country. Higher values indicate better governance ratings. Percentile ranks have been adjusted to account for changes over time in the set of countries covered by the governance indicators. The statistically likely range of the governance indicator is shown as a thin black line. For instance, a bar of length 75% with the thin black lines extending from 60% to 85% has the following interpretation: an estimated 75% of the countries rate worse 13

14 and an estimated 25% of the countries rate better than the country of choice. However, at the 90% confidence level, only 60% of the countries rate worse, while only 15% of the countries rate better. This means that many differences in percentile rank are not statistically significant. This section, as state above, only highlights those differences that are statistically significant, according to Kaufmann et al. Figure Costa Rica scores better than all other Central American countries on Political Stability. Panama scores significantly better than Guatemala. There are no other significant differences among the Central American countries with respect to Political Stability. 14

15 Figure Costa Rica and Panama are better ranked on Government Effectiveness than Honduras, Guatemala and Nicaragua. El Salvador has better Government Effectiveness than Nicaragua. 15

16 Figure Costa Rica and Panama have better Regulatory Quality according to Kaufmann et al. than Honduras and Nicaragua. 16

17 Figure Costa Rica has better Rule of Law than all other Central American countries. Panama has better Rule of Law than Nicaragua, Honduras and Guatemala and El Salvador has better Rule of Law than Guatemala. 17

18 Figure Costa Rica has better Control of Corruption than all other Central American countries, according to Kaufmann et al. El Salvador has better Control of Corruption than Honduras. Comparison with the income category average 21. Costa Rica scores significantly better in Voice and Accountability and Political Stability than its income category average - with no significant differences on the other WGIs. Panama is significantly worse in Regulatory Quality and Control of Corruption than its income category average. El Salvador is significantly better in Voice and Accountability, Regulatory Quality and Control of Corruption than its income category average. Guatemala is better in Regulatory Quality and worse in Political Stability and Rule of Law than its income category average. Nicaragua is worse in Government Effectiveness and Honduras in Regulatory Quality than the income category average. Figure 11 illustrates the comparison with the income category average for Rule of Law. The charts for the remaining five WGIs are in the appendix. 18

19 Figure 11 How the Central American countries compare to other Latin American countries 22. As shown in the appendix, no generalizations can be made about the performance of Central American countries with respect to the other Latin American countries: while some countries like Costa Rica are in the top group of all Latin American countries on many indicators, others like Honduras tend more towards the bottom group, with several Latin American countries like Venezuela or Paraguay frequently scoring worse (see appendix). This means that some Central American countries score below and some above the Latin American Regional average, as illustrated in Figure 12 for Voice and Accountability. 19

20 Figure Many differences between the Latin American countries are not statistically significant, so that, for instance, the performance of 11 other Latin American countries is not significantly different from that of Honduras on Voice and Accountability. 24. However, many statistically significant differences can still be found: Chile, Uruguay, Costa Rica, Panama, Brazil and Argentina score for instance significantly better on Voice and Accountability than Honduras. 25. In comparison to the income category average, many other Latin American countries also do not score very well on Rule of Law, as illustrated in Figure 13. The only country that scores significantly higher than its income category average is Chile. The bad performance in Rule of Law therefore seems to be a common problem in the Latin American region, according to the data of Kaufmann et al. (2007a). 20

21 Figure 13 21

22 Over-time changes in Central American countries 26. According to Kaufmann et al, there are almost no significant changes in the quality of governance in the Central American Countries for any of the WGIs in terms of percentile rank (see appendix). In fact, the only significant change from 1996 to 2006 is an improvement in the Control of Corruption in El Salvador. A word of caution 27. Drawing automatic conclusions from changes over time and across countries in the WGIs about actual changes and differences in the quality of governance should be avoided, as this section explains. The rule of thumb that confidence intervals should not overlap is a good start, but not sufficient. Different criteria across countries and over time 28. The criteria on which countries are ranked differ across countries and over time because the composition of sources and the weights assigned to them differs across countries and over times. Figure 4 (see appendix) illustrates the change in sources and therefore criteria for rating countries over time. Figure 14 illustrates how the weights and sources differ across countries and Figure 15 illustrates how the weights and sources differ over time. Figure 14 Weights 2004, CC (as originally published) 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% BELIZE CUBA ROMANIA SENEGAL EL SALVADOR wmo wcy usd qlm prs prc pia mig lob gcs gal fht eiu dri cdu ccr bri bps asd agi afr adb Source: Anderson, James; Comments to the first draft note, Dec, For an explanation of the legend please refer to the Annex: Weights 2006 by Source and Worldwide Governance Indicators 22

23 29. Figure 14 shows the weights given to different sources for the 2004 Control of Corruption indicator for El Salvador and the two countries that rank immediately above and below it. There is a striking difference in the mix of sources which means that the implicit definition of what is being measured is different in each country. To illustrate, since weights are simply increased proportionally to make up for missing data, the rating for Belize is based to about 46% on the CPIA (in orange, labeled PIA), while its share is much lower for El Salvador. Larger countries with more sources tend to have more similar sources and therefore similar shares of weights, although still not being identical. 30. The weights also vary over time. Below are the weights for Nicaragua over time, again they have been proportionally increased to make up for missing observations. The 1996 score is particularly weak for all countries, since the number of sources was smaller then. In Nicaragua, 69% of the weight in 1996 went to EIU, while the same source got only 13% weight by Figure % 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Weights over time, CC (Nicaragua, as published in GM7) tpr pia ifd hum ebr asd eiu adb rsf obi msi her gii frh prs ccr bti wcy prc lbo gwp gcs gcb bps afr wmo Source: Anderson, James; Comments to the first draft note, Dec, This different mix of indicators over time and across countries without any conceptual explication also makes the interpretation of the margin of errors unclear: A statistic such as the confidence interval around a sample mean has a very specific interpretation: There exists a population of something specific. We estimate the mean of the population using observations 4 For an explanation of the legend please refer to the Annex: Weights 2006 by Source and Worldwide Governance Indicators 23

24 on that same specific thing, but only for a subset of the population. Since we only have measurements for a sample, we arrive at a confidence interval for the true population mean for that specific thing. The WGIs have different mixes of indicators in different countries and in different years. We don t have different observations on the same thing; we have different observations on different things. What the margin of error means in such a case is unclear. (Anderson, James; Comments to the first draft note, Dec, 2007) Reasons for a change over time 32. An actual change in the level of governance is only one of four possible reasons for a change in a country s rating on the WGIs. 33. A second possible reason is a change in the sources perceptions of the quality of governance without an actual change in the quality of governance. For instance, in December 1996, a string of corruption cases hit Charles Haughey, who had been prime minister in Ireland from 1979 to The International Country Risk Guide, one of the sources for the WGIs, downgraded Ireland several times in the months that followed and during which more and more evidence of the scandals appeared. It is unlikely that the level of actual corruption had changed in Ireland, but the ICRG updated its rating based on newly available information. 34. A third possible reason for a change is that other countries governance ratings changed. Since Kaufmann et al. construct each of their composite indicators in such a way that the average value of the indicator across all countries, worldwide, is always zero and its standard deviation is always one, changes in one country s rating change the ratings of other countries, ceteris paribus. 5 However, percentile ranks have been adjusted to account for changes over time in the set of countries covered by the governance indicators, according to Kaufmann et al. 35. A fourth possible reason is a change in the number and composition of sources and the weights assigned to them from one year to the next. Changes in the components of a composite indicator over time are not necessarily problematic for its comparability over time, if there are substantive reasons for the change (for instance, computers were added to the basket of goods 5 Kaufmann et al. (2007b, p. 3.) claim to have found evidence from individual sources that world averages of governance are not much and that this evidence allows them to interpret the relative changes in country scores on our aggregate indicators, or groups of countries scores, as absolute changes. We do not find this argument persuasive for the following reasons: i) The evidence that global averages do not change over time is based on the assumption that all underlying sources are comparable over time. However, many underlying sources used to construct the KKZ indicators do not allow for reliable comparisons of levels of governance over time: To illustrate, rating changes in expert-based systems do not always reflect a belief that actual conditions have changed, but often are intended to correct a previous year s rating that in retrospect appears too high or too low. The documentation of the Heritage Foundation s Property Rights Index, one of the sources for Rule of Law, frequently mentions the availability of new data as the core reason for a change in a country s rating 5 and World Bank staff involved in the construction of the CPIA affirmed that new ratings often correct last years ratings. ii) There is no evidence to the contrary either, i.e. there is no empirical basis on which to conclude that there are not trends or other important types of variation in the global quality of governance over time. iii) Not only would global averages need to be constant, so too would standard deviations need to be constant, to allow for reliable comparisons of levels of governance over time on the basis of the KKZ indicators, yet there is also no reason to believe that the standard deviations are constant. 24

25 used to calculate the Consumer Price Index, because consumptions patterns change). However, changes in the WGI sources are not explained beyond more data availability. 36. An illustration of the sensitivity of the ratings to the inclusion of new sources is the case of the WGI "Regulatory Quality" in El Salvador and Bolivia from 1998 to 2005, as highlighted in Figure 16. While Kaufmann et al. found in their 2006 publication Governance Matters V a statistically significant deterioration of "Regulatory Quality" from 1998 to 2005 (see Figure 16a) in both countries, they do not find one in the recent update of their work (Kaufmann et al., 2007a, Governance Matters VI, see Figure 16b) for the same period. The difference in the two graphs is mainly due to a difference in the rating for the year 1998, due to a change in sources from Governance Matters V to VI (see Table 1). In Figure 16a), the rating for 1998 is based on three sources for El Salvador and five sources for Bolivia, while in Figure 16b) it is based on six sources for El Salvador and seven sources for Bolivia. The additional sources change the percentile rank for 1998 dramatically, from 94.1 to 78 in the case of El Salvador, and from 78.8 to 60 in the case of Bolivia. Due to the change in rating and confidence interval for 1998, the over time change from 1998 to 2005 is smaller and no longer statistically significant. Figure 16a) "Regulatory Quality" 1998 and 2005 in El Salvador and Bolivia, Governance Matters V Source: taken from the WGI website based on data in Kaufmann et al. (2006) 25

26 Figure 16b) "Regulatory Quality" 1998 and 2005 in El Salvador and Bolivia, Governance Matters VI Source: taken from the WGI website based on data in Kaufmann et al. (2007a) Table 1a) Sources for "Regulatory Quality" 1998 El Salvador and Bolivia in Governance Matters V Data for figure 16a) El Salvador "Regulatory Quality" 1998 (score of 1.31 on a scale from -2.5 to +2.5) Source Publication Type Source s rating (scale from 0 to 1) 1 Heritage Foundation/Wall Street Journal 2 World Bank 3 Political Risk Services Economic Freedom Index Country Policy and Institutional Assessments International Country Poll Poll Not public Poll Risk Guide Bolivia "Regulatory Quality" 1998 (score of 0.8) 1 Global Insight Global Risk Service Poll Heritage Foundation/Wall Street Journal 3 World Bank 4 Political Risk Services 5 World Bank Economic Freedom Index Country Policy and Institutional Assessments International Country Risk Guide World Business Environment Survey Poll Poll Not public Poll Survey

27 Table 1b) Sources for "Regulatory Quality" 1998 El Salvador and Bolivia in Governance Matters VI Data for figure 16b) El Salvador "Regulatory Quality" 1998 (score of 0.74 on a scale from -2.5 to +2.5) Source Publication Type Source s rating (scale from 0 to 1) 1 Economist Country Risk Service Experts 0.55 Intelligence Unit 2 Global Insight Business Conditions and Experts 0.63 Risk Indicators 3 Heritage Index of Economic Experts 0.8 Foundation Freedom 4 Political Risk International Country Experts 0.83 Services Risk Guide 5 World Bank Country Policy and Experts Not public Institutional Assessments 6 World Economic Forum Global Competitiveness Report Survey 0.45 Bolivia "Regulatory Quality" 1998 (score of 0.35) 1 Economist Country Risk Service Experts 0.6 Intelligence Unit 2 Global Insight Business Conditions and Experts 0.63 Risk Indicators 3 Global Insight Global Risk Service Experts Heritage Index of Economic Experts 0.7 Foundation Freedom 5 Political Risk International Country Experts 0.83 Services Risk Guide 6 World Bank Country Policy and Institutional Assessments Experts Not public 7 World Economic Forum Global Competitiveness Report Survey Given the complexity of the WGIs and the many reasons for changes over time, it is difficult to attribute any changes in the WGIs to a specific reform. Nor does the success or failure of a reform necessarily lead to any changes in the WGIs. The criteria and norms behind the WGIs 38. Stating that one country has better governance than another country or that a country improved or worsened its governance on the basis of an indicator means endorsing its criteria and norms. This means that awareness of the norms and criteria is crucial. 39. It is important to remember that all rating systems are normative by their very nature. They imply a judgment if they rate one country better than another country. To illustrate, for an overall "Rule of Law" indicator, several choices have been made: Should indicators that assess property rights and/or indicators that assess crime be included? How much weight should be assigned to an indicator of the murder rate compared to an indicator of the respect of property rights? Is access to justice for the poor more important than exposure to street crime in the 27

28 diplomatic quarters of a city? How much weight should therefore be assigned to each indicator? Depending on the choices made, ratings of the overall quality of the "Rule of Law" can then emphasize the perspectives of the private sector, the poor, international investors, low or highlevel crime, or rate formal law systems better than informal ones. 40. Unfortunately users of the WGIs will not find an easy answer to the norms and criteria underlying the WGI ratings. The reason is that they are not based on a normative definition or concept of what constitutes good or bad governance and how components of governance interact Instead, the sources brought together in a single WGI give normative meaning to the concept of governance. However, nobody, including the authors themselves, can define these norms, which also differ across countries and over time as mentioned before and illustrated in Figure 4 in the appendix and Figure 14 and 15. The assumptions made about what governance in what combination is good in each country are therefore implicit and change over time and across countries (see Box 2). 42. There is also considerable overlap between the six indicators, some of which is unavoidable. For instance, criminal corrupt behaviour both undermines the Rule of Law and the Control of Corruption. Tax policy and administration, including tax evasion, are covered by numerous indicators with most assigned to Regulatory Quality but others assigned to Government Effectiveness. Box 2 What norms and criteria underlie the WGIs? Given the absence of a normative concept of governance underlying the WGIs, we can only speculate about the normative content of the WGIs in looking at the sources and the names of indicators underlying each source s rating as well as the weights assigned to sources with wide country coverage. Regulations to protect the environment and labour rights tend to be interpreted as unfriendly to markets by the sources that carry the most weight in the aggregation process. Whereas one may or may not agree with this interpretation, the problem is that most users are not aware of this bias. For example, Kaufmann et al. include the indicator Environmental regulations hurt competitiveness from the World Economic Forum s Executive Opinion Survey, but they do not include several questions that give high ratings to countries with a high standard of environmental protection. Users are generally not aware of this omission. While the list of sources used to produce the KKZ indicators (see Appendix) appears to be reasonably diverse and representative of different stakeholders ranging from population surveys to expert assessments and enterprise surveys the aggregation procedure used to calculate the composite indicators assigns less weight to sources that differ from the majority, as explained previously. The result of this procedure is effectively to give much more weight to expert assessments and enterprise surveys than to population surveys to the point that population surveys carry practically no weight in the composite indicators. To illustrate, Gallup s 6 Government output at the macro level cannot be considered a simple aggregation of government outputs at the micro-level. In other words, achieving one outcome interacts with achieving other outcomes (van de Walle, forthcoming) 28

29 World Poll that asks citizens about their exposure to crime gets zero weight for the WGI Rule of Law, whereas Global Insight Business Risk and Conditions, a commercial business information provider from Boston that measures the crime risk to businesses, gets the third highest weight. An example of a Global Insight indicator is How much of a threat businesses face from crime such as kidnapping, extortion, street violence, burglary and so on. These problems can cause major inconvenience for foreign investors and require them to take expensive security precautions. There are two interconnected reasons behind the low weights given to household surveys: first of all, there are fewer cross-country comparable household surveys and they provide data for fewer countries, because they are expensive and because there are few opportunities to exploit the data commercially. Kaufmann et al. use most cross-country comparable governance indicators available and commercial interests drive a major part of this supply of governance indicators. Secondly, while the authors are not responsible for the low number of household surveys available, they could have corrected for this selection bias in the aggregation process. Instead, they chose to use an aggregation technique which gives less weight to perceptions that diverge from the dominant majority than to perceptions that conform to the majority view. Household surveys diverge from the majority of expert perceptions, and therefore, get low weights. This divergence is confirmed by mirror surveys in francophone Africa undertaken by DIAL which show that the assessments of the quality of local governance given by experts differ substantially from those given by respondents to local population surveys 7. The reason for this divergence is not necessarily that either household surveys or expert assessments are wrong, but could simply be that they assess different aspects of governance. To come back to our previous example, the crime rate and protection (or abuse) by the police and judiciary are not necessarily at the same level for the poor parts of a city and the diplomatic and business parts. However, Kaufmann et al. assume that the discrepancy between Gallup s rating and the majority of expert ratings is due to its high measurement error and discard the possibility that it measures something different. 8 7 Razafindrakoto and Roubaud (2006) 8 Kaufmann et al. (2007b) state that indexes produced with alternative weighting systems are very highly correlated with their original WBI indicators, and conclude that criticisms of their weighting methods are of little importance: The rankings provided by our aggregate indicators are quite robust to alternative weighting schemes (Kaufmann et al., 2007b, p. 13). However, neither of these two methods used to make their point corrects for the fact that household surveys are in the minority, nor do they take into account that those sources that rate countries different from the majority of sources may actually contain very useful information about the quality of governance. This is most obvious with the first method Kaufmann et al. use, a simple average of all sources called equal weighting. In fact equal weighting is very unequal: since there are many more expert assessments and firm surveys than surveys of individuals available, the weight of household surveys is still negligible. Similarly, in the second weighting method, weighting according to groups, household surveys are put together with firm surveys into one group which receives one fifth of the weight in total, which reduces the weight of household survey to a not-even specified fraction. Given that household surveys show very different results, it is obvious that the claim that different weighting systems do not lead to substantially different results cannot be supported. 29

30 This sample bias in favour of business-oriented perceptions is not unique to the KKZ indicators. Transparency International s Corruption Perceptions Index (CPI), which draws on many of the same sources as KKZ s Control of Corruption indicator, is characterised by a similar bias. As one of Transparency s founders and former head of research has noted, Of the 17 different institutions providing data for the CPI since 1998, only two do not have a private-sector bias This homogeneity generates a serious sample bias, which is a genuine failing of the CPI. The sample is not only private sector oriented, it is also overwhelmingly male and economically well off. Effectively, this means that this most influential of indices ignores the experiences and perspectives of most women, and of the poor and disenfranchised. It also means that the interests of unofficial businesses, which employ the overwhelming majority of the population in poor countries, are ignored (Galtung, 2005). 43. Since sources measure different things and assign different ratings to countries, different weighting systems would also lead to different ratings, as illustrated for Regulatory Quality Figure 17 shows the overall results of the rating for five Central American countries with the same sources. Table 2 shows how each of the ten sources rated the five Central American countries. Figure 17 30

31 Table 2 "Regulatory Quality" 2006 in Costa Rica, Guatemala, Honduras, Nicaragua and Panama Costa Rica Guatemala Honduras Nicaragua Panama Percentile Rank Source Source rating (on a scale from 0 to 1) Bertelsmann Transformation Index Economist Intelligence Unit Global Insight Business Conditions and Risk Indicators Global Insight Global Risk Service Heritage Foundation Index of Economic Freedom IFAD Rural Sector Performance Assessments Merchant International Group Gray Area Dynamics Political Risk Services International Country Risk Guide World Bank Country Policy and Institutional Assessments Not public Not public Not public World Economic Forum Global Competitiveness Report Scores from different individual indicators are not directly comparable with each other since the different data sources use different units and cover different sets of countries. The data from the individual indicators are further rescaled to make them comparable across data sources before constructing the aggregate governance indicators. See Aggregating Governance Indicators (Kaufmann et al., 1999) for details. 44. Guatemala comes third out of the five Central American countries with the same sources. However, only two sources rank it third. Three sources rank it first (World Economic Forum Executive Opinion Survey, IFAD Rural Sector Performance Assessments, Political Risk Service International Country Risk Guide,), one ranks it second, two rank it fourth and one ranks it fifth. One of the sources that rank it third, Global Insights Business Conditions and Risk Indicators, has more weight than the three sources taken together that rank Guatemala first. If considerably more weight were given to those sources that rank Guatemala first, Guatemala s position on the rating would improve. 45. Panama comes second for Regulatory Quality, after Costa Rica. However, out of the 9 sources we have data for, only four ranks Panama behind Costa Rica. Two give the same score to Costa Rica and Panama and three rank Panama higher than Costa Rica. Taken together, the four sources that rank Costa Rica higher than Panama have almost double the weight than the three sources that rank Panama higher than Costa Rica. Again, with different weights, Panama would improve its ranking. 46. The high divergence in the sources ratings is reflected in the high margins of error: Guatemala s 90% confidence interval overlaps with the confidence interval of all other four countries. And Panama s confidence interval overlaps with Costa Rica and Guatemala (see Figure 17). Again, this reinforces the warning that rating countries on the WGIs, especially - but not only - where the confidence intervals overlap, is misleading. 31

32 47. Furthermore, rating countries on true governance implies that there is a best or ideally performing government (van de Walle, forthcoming). One may wonder about the nature of the opposite, untrue governance, and whether a single system of desired governance outcomes and priorities should be promoted in the developing world. Meisel (2004) explains why Anglo- Saxon glasses do not help to explain the rapid growth in the Asian economies and offers a new model of Governance Cultures, helpful for developing countries today. There will be further elaboration on the debate about appropriate governance for developing countries and the challenges involved in measuring it, in the final section of this paper. Purpose 3: Identify reforms 48. Are the WGIs sufficiently disaggregated to assist in identifying which reforms are most needed in a given country to improve its quality of governance? While they can very broadly indicate that there is something wrong in a country, opaque summary measurements do not lend themselves as tools to identify a country s specific weaknesses and the specific reforms necessary to overcome this weakness. 49. The WGIs give little guidance to local stakeholders or others as to what concrete actions they can or ought to take to improve the quality of local governance. For instance, an indicator for Rule of Law measures how secure business people feel about their property, but it tells us very little about what makes them feel that way. The indicators are used mainly as a decision tool for outsiders, who wish to judge or compare countries, but are not action-oriented for stakeholders in the countries that are being judged and compared. There are at least three reasons why countries cannot draw actionable conclusions from the WGIs: 50. First of all, the sheer number and diversity of indicators, produced by different sources that are incorporated into a single WGI make it very difficult to understand a country s rating, as illustrated in Figure 4 in the appendix for Regulatory Quality. Understanding a change in a country s rating over time or a cross-country difference is complicated, because the composition and weights of primary sources vary across countries and over time (see Figure 4, 14 and 15). This difficulty in understanding the meaning of a country s rating is all the greater because some of the underlying indicators used as inputs are themselves very broad and imprecise or lacking transparency. To illustrate, it is not clear what exact criteria the Economist Intelligence Unit (EIU) uses for rating countries on the indicator Excessive Protections, one of the sources for Regulatory Quality (see Figure 4). When are protections necessary and when are they excessive according to the EIU? Many sources unfortunately provide only headings for their indicators, with no definition or documentation of how their assessments were made. 51. Secondly, many of the indicators underlying each source s ratings (the 2nd layer in the pyramid) are not disclosed to the public (see Figure 4 in the appendix). Kaufmann et al. publish only the overall rating of each primary source in rescaled form. Access to some of the indicators of the 2nd layer would be very expensive. For instance, access to the source Global Risk Data from Global Insight, cost $ in 2004 ( inquiry to Global Insight in 2004). 52. Thirdly, the WGIs lack a clear theory of good or bad governance that would explain the meaning of the indicators. 32

33 Purpose 4: Research 53. The WGIs can support or challenge common perceptions in the six broad governance categories they cover. To illustrate, Chile has better governance than most other developing countries according to the WGIs and common perceptions. However, since the weighting system gives high weights to the opinions of foreign experts, it is unlikely that many of the WGI results differ considerably from accepted stylized facts among foreign experts. They may differ though from the perceptions of the population or the government. 54. The WGIs do not allow for identifying sequencing issues in specific reforms and relationships between improvements in specific governance areas like public expenditure management and service delivery. Purpose 5: World Bank credibility and country dialogue 55. Kaufmann et al. emphasize the importance of looking carefully at the underlying data sources and complementing their indicators with more in-depth and country-specific analysis. Whether the WGIs enhance World Bank credibility in country dialogue depends both on the way they are presented and the receptiveness of a country to the views of outside experts of their country. Interviews with World Bank staff in several countries revealed very different experiences (Arndt and Oman, 2008). While some staff reported finding the WGIs very useful as a starting point for dialogue, others reported hostile reactions. 56. One should never forget that most perception indicators are likely to exhibit a time lag when reflecting actions taken by the government to remedy some governance problems and that the results of reforms in some areas which are not highly weighted in the WGIs may never show up in the WGIs, even if these reforms are successful (Van de Walle, 2006). Given that a WGI like Government Effectiveness comprises such diverse indicators as quality of infrastructure and public health and education services as well as indicators of red tape with changing weights assigned to them, it is unlikely that even a very successful reform in an area like tax administration will show up in the rating. Furthermore, changes may have occurred despite government intervention, or even just by chance (Bonar Blalock, 1999, Steven van de Walle, forthcoming, Pollitt, 2000). According to Bertok, Hall, Kraan, Malinska, Manning and Matthews (2006), outcomes cannot usually be simply attributed to government actions. The media often uses changes in the WGI ratings to shame or applaud governments (Arndt and Oman, 2008). World Bank staff may exercise more care than the media in order to avoid blaming governments for results they are not responsible for or which are offset by good governance in areas not represented in the WGIs. 57. The WGIs should certainly be presented not as the truth, but as a summary measurement of the views of a wide range of experts. Experts views matter, because they partly determine investment decisions, but so do the perceptions of households and firms within the country: Surveys of firms and households may be better grounded in country realities. The views of respondents matter, because they are able to act on their beliefs. If they believe the courts are highly corrupt, they will avoid seeking legal recourse through the courts and instead choose arbitration or informal means of 33

34 settling disputes. While governments may discount outsiders views, citizens and firms views matter 9 (World Bank, 2008). 58. Views often differ considerably: for instance, results of the in-country business enterprise surveys frequently do not match the results of the WGIs. Disentangling the views of outside and in-country experts, firms based in the country and household surveys of the quality of governance in the country is certainly a useful way of starting a discussion. The next section will give recommendations on how staff can complement or substitute the WGIs, depending on the specific country situation, with a range of other, more specific, indicators to start off country dialogue. 9 Of course also households and firms perceptions do not always fully reflect the reality on the ground (see Kampen et al., 2006) 34

35 IV. SO WHERE DOES THAT LEAVE US? CONCLUSION AND RECOMMENDATIONS 59. This note has attempted to test the WGIs against a set of possible purposes for governance indicators. The result is nuanced: 1. Do the WGIs identify whether countries share similar governance characteristics and qualities and therefore assist in drawing lessons for peer learning? They do not identify whether countries have similar governance systems. They do give an indication on how countries compare to each other with respect to the quality of their governance - within certain limitations due to differences in sources across countries and related conceptual problems. 2. Do they provide a time series to identify whether countries are improving? Over-time changes in the WGIs can indicate an actual changes in the quality of governance, but can also be due to changes in the composition of underlying sources, changes in underlying sources criteria and methodology, changes in other countries ratings and changes in the perception of governance, independent of actual changes in governance. A change in the WGIs should therefore be the starting point of a deeper analysis, not the endpoint. 3. Are they sufficiently disaggregated to assist in identifying which reforms are most needed in a given country to improve its quality of governance? Generally not. 4. Are they useful for upstream research purposes? They can indeed challenge or support common broad perceptions, but do not allow for identifying sequencing issues in specific reforms and relationships between improvements in specific governance areas 5. Do they reinforce the Bank's credibility as a source of expertise and so enhance country level dialogue? They can, if they are carefully presented as a first cut of experts perceptions of the quality of governance and not as the final word. Alternatives and complements 60. While the use of governance indicators is concentrated around a few summary measurements, there is actually a plethora of meaningful indicators available. Some are already available for direct use (section Working with what we have ) and others, more refined indicators, are still in their infancy, with data for fewer countries and years Progress towards more actionable indicators (section Moving ahead creatinga set of more robust indicators ) that can help countries identify areas to reform and to monitor progress over time is not only limited by resources and technical capacity: constructing a set of actionable indicators as a reform-tool for policy-makers for every area of governance requires a deeper understanding of the governance priorities in developing countries than the state of current research allows for. Research also points to strong individual differences in the governance of the growth-miracle countries in the past decades a challenge for the development of cross-country comparable actionable indicators (section Conceptual challenges and the debate about actionworthiness ). 35

36 Working with what we have 61. In our present state of knowledge, the WGI indicators should be used, when politically feasible, as a point of entry to a discussion rather than as a normative framework for implicitly reprimanding the poor performers. Producers of the WGIs could assist in rendering their indicators more useful for country-dialogue, by further raising the transparency of their indicators and disclose more data and information to better convey the meaning and limitations of the WGIs. 62. More concretely, the authors of the WGIs could explain why they choose certain indicators and not others from each source for inclusion in each WGI. An explanation from their sources of the criteria they used to rate countries would also be very useful. For instance, the authors could ask the Economist Intelligence Unit to explain what they mean by Excessive protections. The authors of the WGIs may also extend their very user-friendly webpage to generate graphs like Figures 14 and 15 to show the composition of sources and the weights attributed to it for each country s rating. The authors could highlight more clearly, as Transparency International does for their Corruption Perceptions Index, that they are a summary measurement of mainly experts perceptions of the quality of governance and not a measurement of true governance. Currently, the WGI authors typically list household and firm surveys first in overviews of their sources, which can create an exaggerated impression of their influence on the indexes. The WGI authors could also advise users more clearly to exercise care in drawing conclusions from changes over time or cross-country differences, beyond the margins of errors. 63. Debate in developing countries can also be encouraged by using a broader array of other existing data, some of which underlie the WGIs and are well-documented on the user-friendly website of the WGIs. Many other user-friendly guides and databases exist to make the wealth of available data easily accessible and comprehensible (IADB, 2007, UNDP, 2007, Metagora, 2007, World Bank Institute, 2007, Besancon, 2003). The Inter American Development Bank s (IADB) web tool DataGob 10, for example, provides comprehensive information about countries results on over 400 indicators, in comparison to other countries and regional as well as income averages, the methodology used to build them and the indicators strengths and limitation. Box 3 gives two examples of indicators that might be useful for country dialogue manual: 36

37 Box 3 Examples of indicators available for Latin America Meritocractic practices in the civil service LAC BRA CHL CRI COL URY MEX VEN ARG BOL PRY DOM ECU PER GTM HND NIC SLV PAN Source: (2006) Informe sobre la Situación del Servicio Civil en América Latina (ed. K. Echebarría). Washington, D.C.:IDB, Dialogo Regional de Política, Red de Gestión y Transparencia de la Política Pública. The original paper is: Longo, Francisco (2002) 'Analytical Framework for the Institutional Assessment of Civil Service System', Washington, D.C.: IDB, Regional Policy Dialogue. Description: Measures the degree to which effective guarantees of professionalism in the civil service are in place and the degree to which civil servants are effectively protected from arbitrariness, politicization and rent-seeking. The indicator has been normalized from 0 to 100 with higher scores indicating greater degree of professionalism and meritocracy. The indicator values are drawn from a series of country studies of civil service systems commissioned by the IDB Regional Policy Dialogue on Public Policy Management and Transparency based on a methodology developed by Francisco Longo. A summary of the results of these studies and the indicators is presented in Koldo Echebarría (ed.). Informe sobre la situación del servicio civil en América Latina. Washington, D.C.: Inter-American Development Bank, (IADB, DataGob, accessed April 2008) Methodology: This indicator is derived from assessments produced by a small group of researchers based in a common methodological framework. To determine ratings for each country, researchers collect and analyze available information, converting data into ratings on the basis of a consistent pattern of evaluation. Generally, indicators produced by a small group of researchers using a common checklist or framework of analysis have the advantage of permitting valid cross-country comparison. In this case they are accompanied by detailed country reports that justify and contextualize numerical ratings. As analytically-based indicators, they also have the advantage of bridging the gap between de jure and de facto institutional arrangements. The framework of analysis used to construct the civil service indicators measures specific institutional arrangements in respect to seven different subsystems related to the management of human resources in the public administration. The conceptual consistency and precision of the indicators allows them to be useful for identifying specific problem areas and guiding reform decisions. The studies aim to be as transparent as possible about the methodology and scoring system used to assign ratings. The IDB civil service indicators do not allow comparisons over time, as studies have been carried out only once in each country and it is not clear when or if they will be repeated. (IADB, DataGob, accessed April 2008) 37

38 LAC CHL Bribes for influencing laws, policies, regulations or decrees URY COL CRI SLV MEX BRA GTM PER PAN ARG DOM NIC VEN PRY ECU BOL HND Source: World Economic Forum. The Global Competitiveness Report. Geneva: Palgrave MacMillan. Description: Average responses of business executives surveyed to the following question (scale of 1 to 7): In your industry, how commonly would you estimate that firms make undocumented extra payments or bribes connected with influencing laws and policies, regulations, or decrees to favor selected business interests? (1= common; 7= never occurs). This question is a part of the World Economic Forum's Executive Opinion Survey carried out for the Global Competitiveness Report. An average of 94 Chief Executive Officers or top-level managers are polled in each country from a sample of companies which includes domestic firms that sell in foreign markets, units of foreign firms that operate in the domestic market and enterprises with significant government ownership (where applicable). (IADB, DataGob, accessed April 2008) Methodology: This indicator is a perception-based indicator derived from responses to the WEF executive opinion survey. Surveys of this type provide valuable information on a broad range of variables for which hard data is typically scarce. They have the advantage of permitting the construction of indicators with a relatively high degree of conceptual precision. The homogeneity of the sample group allows the use of fairly well-specified concepts, and the use of a large set of questions permit the unbundling of broad concepts. These types of indicators have some potential to be used to identify specific problems and to provide some guidance in identifying potential areas for policy intervention. Indicators based on survey data are also useful because they are able to capture unwritten, informal institutional arrangements that can influence the quality of governance in a country. The World Economic Forum (WEF) is recognized a world leader in measuring economic competitiveness. WEF data have been used in peer-reviewed articles, and the methodologies used to construct indicators are subject to internal and external review. Thus, the data are considered to be fairly reliable. However, comparability over time is limited to some extent by the changes in country coverage and structure of the survey over time (including the specific questions asked) (in 2006 there were significant changes in methodology; see report for details). Users should bear in mind that the executive opinion survey captures perceptions of business executives; thus these perceptions are not necessarily representative of those of the broader population. Survey responses may also be unduly influenced by prevailing or past trends in economic conditions and/or by recent events. Finally, users should bear in mind that indicators based on survey data measure perceptions of individuals who inevitably must base their judgements on a limited range of personal experience and factual knowledge. (IADB, DataGob, accessed April 2008) 38

39 64. As noted by the UNDP s user guide (2007) and World Bank s World Development Indicator report (2008), it is crucial to use a set of indicators and to understand differences in results. Every source gives different information and adds a piece to the puzzle of a country s governance. Looking at a country s ratings from transparent sources and the differences between these results can be a very good starting point for a deep country dialogue. 65. One possible reason for differences in results across indicators is that one indicator is wrong. Another reason is that indicators measure different things or simply the experience of different people. Business surveys will be more informative about bribes paid by businesses and the security of companies and their employees than household surveys. However, they will probably not reveal whether ordinary citizens are protected or abused by the police and whether ordinary citizens have to pay bribes for services that should be provided for free. 66. Surveys that ask explicitly about different peoples own experience as opposed to generic questions about a country s level of corruption are, therefore, very useful.. To illustrate, the Governance Diagnostic Surveys from the World Bank Institute are separate surveys of households in different income categories, business managers from different types of firms and public officials. Results for Honduras, for example, indicated that foreign firms report more incidences of bribe demands in the registration of business, properties, and vehicles, and taxation processes. Small firms, on the other hand, face bribe demand from public officials when they apply for import/export permits and phone installation and they report almost no demand for bribes in registration services. Results for Peru indicated a much higher rate of bribery being paid by households trying to obtain a public service in the Andean Sierra region (about 15 per cent) than by households in the tropical Selva region (about 7 per cent). Results such as those cited here from Honduras and Peru must of course be interpreted with care, not least because different respondent groups may have different fears or other motivations to either understate or overstate their true experience, for example. Our point is simply that these country-specific Diagnostic Surveys are very useful for policy makers and others seeking to improve the quality of governance in a given country probably more so than are the WGIs. Governance Diagnostic Surveys are available for the Central American countries Guatemala and Honduras. Moving ahead creating a more robust set of indicators 67. The problem of the lack of specific, meaningful and actionable data concerning governance, or even concerning public management more narrowly conceived, is well known. The OECD has recognized this problem and has launched an initiative to collect disaggregated data on public sector processes in order to address the "persistent problem in public management reform recommendations (namely that) they are rarely based on empirical evaluations, and in practice can owe more to policy fashion than to evidence and with significant over-claiming about best practice." OECD (2007). Similarly, there has been some progress in the development and use of disaggregated and actionable indicators, such as the public financial management indicators developed by the multi-donor Public Expenditure and Financial Accountability (PEFA) initiative. For the Latin American region, the World Bank plans to develop comparative indicators that benchmark the structure and performance of government bureaucracies in the field of audit, PFM, intergovernmental finance and justice. The project is explicitly designed to help Latin American countries benefit from the experience of their peers and OECD countries and will be complemented by demand-driven technical peer assistance. Table 3 from the recent edition of the World Development Indicators highlights some existing actionable indicators. 39

40 Table 3 Selected actionable governance indicators Indicator or objective Public Expenditure and Financial Accountability Assessment, initiated in 2001, measures critical dimensions of open and orderly public financial management systems. OECD Assessment Methodology for Public Procurement Systems, developed over through an Organisation for Economic Co-operation and Development Assistance Committee and World Bank led roundtable and now being piloted, measures compliance, performance, and transparency and integrity of public procurement systems. Open Budget Index, launched in October 2006 by civil society organizations in 59 counties, provides comprehensive practical information to gauge a government s commitment to budget transparency and accountability. Global Integrity Index, launched in 2002 by the Washington, D.C., based Center for Public Integrity and a new independent nonprofit called Global Integrity formally started in 2005, assesses the existence and effectiveness of anticorruption mechanisms that promote public integrity. The index evaluates the existence of laws, regulations, and institutions; their implementation; and the access average citizens have to those mechanisms. Source: World Bank (2008), World Development Indicators, p. 262 Nature and number of indicators 28 high-level indicators that capture six dimensions of public financial management. 12 indicators with 54 subindicators in four broad areas: legislative and regulatory framework, institutional framework and management capacity, procurement operations and market practice, and integrity and transparency. 122 items that assess public availability of key budget documents, quality of information, and timeliness of dissemination. More than 290 discrete integrity indicators generate the index, which is organized into six broad categories. Country coverage 67 completed, of which 26 are publicly available. 22 countries participating in pilot program; reports available online for in 2006; 88 targeted and 80 expected for countries in 2004, 41 in 2006, 48 in 2007, 33 assessed at least twice. 68. The Global Integrity Index, for example, assesses the existence, effectiveness, and citizen access to key national-level anti-corruption mechanisms used to hold governments accountable. It is generated by aggregating more than 300 Integrity Indicators systematically gathered for each country covered. For the Global Integrity Index 2007, those indicators comprised more than 15,000 peer-reviewed questions and answers scored by in-country experts in the summer of Several rounds of review are conducted at the international level to ensure that crosscountry comparisons are valid. In addition, all assessments are reviewed by a country-specific, double-blind peer review panel comprising additional local and international subject matter experts. All the underlying data and criteria used to rank countries are freely accessible to the public. 40

41 Conceptual challenges and the debate about actionworthiness 69. Quantifying the quality of governance in a transparent and specific way and trying to make indicators actionable forces development analysts to face the limits of their knowledge. Deciding on a set of actionable indicators, useful for reform purposes in developing countries, implies knowledge about what is actionworthy 70. The WGIs summarize all available data and, therefore, make all assumptions about what the priorities of governments should be implicit. Consequently, all the contradictions and disagreements about what governance fosters development or contributes to other goals are covered up. The crucial vagueness of the concept of good governance (Doornbos, 2003; Boas, 2001) helped to put the concept on the agenda (Bovaird and Loeffler, 2003), but does hinder adequate measurement and rigorous analysis (van de Walle, forthcoming). 71. In trying to construct specific actionable indicators, producers and those choosing indicators are necessarily faced with the whole messy question of what governance is good for developing countries: given limited resources, both in terms of money and time, what should governments priorities be? Should basic services be delivered everywhere in the country or should costs be cut? Is it more urgent to provide a secure environment for big companies and their employees or to reduce the crime rate in the poor parts of the city? Should Western-style public management systems be implemented in every developing country? Should priorities be the same in every country? If not, in what context does what governance reform strategy work? 72. Researchers and practitioners worry that the current governance agenda is overloaded (Grindle, 2007) and that it can divert attention away from other, more urgent problems. Rodrik (2008) argues for second-best institutions instead of the current overambitious governance agenda. Andrews (2008, p. 2) argues that the good governance picture of effective government is not only of limited use in development but also constitutes a threat, promoting isomorphism, institutional dualism, and flailing states and imposing an inappropriate model of government that kicks away the ladder today s effective governments climbed to reach their current states. 73. Given all the deep questions behind the development of actionable indicators that touch the very heart of development, it is not surprising that progress towards actionable indicators is slow. However, this may not be such a bad thing after all: A theory of government is needed before we measure government effectiveness or propose specific models of what government should look like. Such theory should address basic questions that center on the fit of different governing systems to different contexts (Andrews, 2008, p.2). The questions and conflicts arising from the process of developing transparent actionable indicators can serve as a basis for debate, together with more qualitative research, about what governance is conducive to development, depending on the country context. It is much better to have this debate now than to start (as happened with so many other development fashions) implementing governance reforms on a large scale that, despite good intentions, might hinder country development more than they help it. 41

42 42

43 V. REFERENCES: Andrews, M. (2008), Are one-best-way models of effective government suitable for developing countries?, KSG Faculty Research Working Paper , Cambridge: Harvard University Arndt, C. and Oman, C. (2008), "The Politics of Governance Ratings". Original draft presented at the IPMN Workshop, Ranking and Rating Public Services, Worcester College, Oxford, 7-9 August Arndt, C. and Oman, C. (2006), Uses and Abuses of Governance Indicators, Development Centre Studies, Paris: OECD Bertok, J., Hall, J., Kraan, D.-J., Malinska, J. Manning, N. and Matthews, E. (2006), Issues in outcome measurement for government at a glance. OECD Project on Management in Government Technical Paper 3, Paris: OECD. Besançon, M. 2003, "Good Governance Rankings: The Art of Measurement.", World Peace Foundation Reports, vol. 36, Cambridge: WPF Boas, M. (2001), Multilateral Development Banks and Sustainable Development: the Strategy of Depolicitisation in F. Biermann, R. Brohm and K. Dingwerth (eds) Proceedings of the 2001 Berlin Conference on the Human Dimensions of Global Environmental Change Global Environmental Change and the Nation State. Potsdam: Potsdam Institute for climate impact Research, pp Bonar Blalock, A. (1999), Evaluation Research and the Performance Management movement. From estrangement to useful integration, Evaluation. 5:2 pp Bovaird, T. and Loeffler, E. (2003), Evaluating the Quality of Public Governance: Indicators, models and Methodologies, International Review of Administrative Sciences. 69:3 pp Doornbos, M. (2003), Good Governance: the metamorphosis of a policy metaphor, Journal of International Affairs. 57:1 pp.3-17 Grindle, M.S "Good Enough Governance Revisited." Development Policy Review 25(5): Inter-American Development Bank (2007), Datagob: Governance Indicator Database. Available online at (accessed April 2008) Kampen, J. K., Van de Walle, S., and Bouckaert, G. (2006), Assessing the relation between satisfaction with public service delivery and trust in government: The impact of the predisposition of citizens toward government on evaluations of its performance, Public Performance and Management Review, 29 (4):

44 Kaufmann, D., A. Kraay and Zoido-Lobatón, P. (1999), Aggregating Governance Indicators, World Bank Policy Research Working Paper, No. 2195, Washington, D.C.: World Bank Kaufmann, D., A. Kraay and M. Mastruzzi (2007a), "Governance Matters VI: Governance Indicators for ", Washington D.C.: World Bank Institute., Available online at Kaufmann, D., A. Kraay and M. Mastruzzi (2007b), The Worldwide Governance Indicators Project: Answering the Critics, World Bank Policy Research Working Paper 4149, March 2007 Kaufmann, D., A. Kraay and M. Mastruzzi (2006), "Governance Matters V: Governance Indicators for ", Washington D.C.: World Bank Institute. Knack (2007), Measuring Corruption: A Critique of Indicators in Eastern Europe and Central Asia, Journal of Public Policy, Cambridge University Press Metagora. Inventory of Initiatives Aimed at Measuring Human Rights and Democratic Governance. Available online at (accessed June 2007). OECD (2007). "Towards Better Measurement of Government (OECD Working Papers on Public Governance, 2007/1)" ( OECD. Paris Razafindrakoto, M. and Roubaud, F. 2006, "Are International Databases on Corruption Reliable? A Comparison of Expert Opinion Surveys and Household Surveys in Sub-Saharan Africa." Paris: DIAL. Rodrik, D. (2008), Second best institutions, Harvard University. UNDP (2007), Governance Indicators: A Users Guide. Avilable online at (accessed June 2007) Van de Walle, S. (2006), The state of the world s bureaucracies, Journal of Comparative Policy Analysis. 8:4 pp Van de Walle, S. (forthcoming), International comparisons of public sector performance: how to move ahead?, Public Management Review. World Bank (2007), "Strengthening World Bank Group Engagement on Governance and Anticorruption", World Bank, Washington DC. Pollitt, C. (2000), Is the emperor in his underwear? An analysis of the impact of public management reforms, Public Management Review. 2:2 pp

45 Pollitt (forthcoming), Simply the best? The international benchmarking of reform and good governance in J. Pierre and P. Ingraham (eds.) Public sector administrative reform and the challenges of effective change, McGill-Queens University Press. World Bank Institute (2002), Governance and anti-corruption in Honduras: an input for action planning, Analysis prepared by the World Bank Institute at the request of the Government of Honduras for discussion with the Consejo Nacional Anti-corrupción World Bank Institute, Governance Data: Web-Interactive Inventory of Datasets and Empirical Tools, available online at (accessed June 2007) 45

46 46

47 ANNEXES Differences between the Central American countries Performance of Central American Countries on the WGIs by Country Indicator Country Sources Year Rule of Law Control of Corruption Regulatory Quality Government Effectiveness Political Stability Voice and Accountability Rule of Law Control of Corruption Regulatory Quality Government Effectiveness Political Stability Voice and Accountability Rule of Law Control of Corruption Regulatory Quality Government Effectiveness Political Stability Voice and Accountability Rule of Law Control of Corruption Regulatory Quality Government Effectiveness Political Stability Voice and Accountability Rule of Law Control of Corruption Regulatory Quality Government Effectiveness Political Stability Voice and Accountability Rule of Law Control of Corruption Regulatory Quality Government Effectiveness Political Stability Voice and Accountability COSTA RICA COSTA RICA COSTA RICA COSTA RICA COSTA RICA COSTA RICA EL SALVADOR EL SALVADOR EL SALVADOR EL SALVADOR EL SALVADOR EL SALVADOR GUATEMALA GUATEMALA GUATEMALA GUATEMALA GUATEMALA GUATEMALA HONDURAS HONDURAS HONDURAS HONDURAS HONDURAS HONDURAS NICARAGUA NICARAGUA NICARAGUA NICARAGUA NICARAGUA NICARAGUA PANAMA PANAMA PANAMA PANAMA PANAMA PANAMA Percentile Rank Governance Score Standard Error (0-100) (-2.5 to +2.5) th-100th Percentile 75th-90th Percentile 50th-75th Percentile 25th-50th Percentile 10th-25th Percentile 0th-10th Percentile 47

48 Performance of Central American Countries on the WGIs by Indicator Country Sources Year Percentile Rank Governance Score (0-100) (-2.5 to +2.5) Standard Error Control of Corruption Rule of Law Regulatory Quality Government Effectiveness Political Stability Voice and Accountability COSTA RICA EL SALVADOR GUATEMALA HONDURAS NICARAGUA PANAMA COSTA RICA EL SALVADOR GUATEMALA HONDURAS NICARAGUA PANAMA COSTA RICA EL SALVADOR GUATEMALA HONDURAS NICARAGUA PANAMA COSTA RICA EL SALVADOR GUATEMALA HONDURAS NICARAGUA PANAMA COSTA RICA EL SALVADOR GUATEMALA HONDURAS NICARAGUA PANAMA COSTA RICA EL SALVADOR GUATEMALA HONDURAS NICARAGUA PANAMA Source: taken from

49 Comparison with the income category average 49

50 50

51 51

52 52

53 53

54 How the Central American countries compare to other Latin American countries 54

55 55

56 56

57 57

58 58

59 59

60 Changes over-time in Central American countries 60

61 61

62 62

63 63

64 64

65 65