GOVERNANCE INDICATORS: AN OVERVIEW

Transcription

1 16/7/02 GOVERNANCE INDICATORS: AN OVERVIEW Introduction We need governance indicators in order to measure performance in developing countries and to track change over time. The importance of measuring governance increases with the trend (in DFID and elsewhere) towards performance based aid allocation and is relevant to our desire to maximise aid effectiveness 1. We also need governance indicators to help identify differences between policy environments and to design strategies and programmes accordingly. To date there is little agreement on how governance should be measured, despite the broad consensus that governance is important for development and growth. The difficulty arises from the multi-faceted and crosscutting nature of governance issues as well as its intrinsically political nature. It is also not clear which facets of governance matter for poverty reduction, when and why. There are however various governance indicator frameworks in circulation, and renewed efforts driven by the OECD DAC, to make progress in finding and agreeing on credible governance indicators. This paper sets out the characteristics, strengths and weaknesses of some of the existing frameworks of governance indicators. It also describes one framework that has been used directly for determining aid allocations in a way that gives priority to governance. In conclusion it offers a favoured set of indicators, which if used in tandem would provide a broad picture of governance, and would be capable of registering change over time. Methodological features of governance indicators There are important differences between types of governance indicators that determine their usefulness for different purposes. 2 The key features to note are: - whether the indicator measures performance (i.e. is normative) or process (i.e. is descriptive). Freedom House s survey of Freedom in the World measures performance (is there more or less freedom than before?) whereas the Database of Political Institutions (DPI) 1 Research by Collier and Dollar explores the circumstances in which aid allocations are optimal. 2 See annex 1 for full list of performance indicators 1

2 describes the variables making up a country s political institutions, but attaches no value to one or other arrangement. - whether the indicator is objective or subjective. Objective indicators are measured in the same way, with clearly defined concepts from year to year and across countries. Subjective indicators generally capture the views of experts or organisations, or are based on surveys of investors or the public, leaving room for cultural or ideological bias. Provided data is available and measured in the same way there should be less error in objective data than subjective data and possibly less controversy. However, subjective indicators can sometimes be more meaningful. For example subjective assessments of democracy may be more accurate than voter turnout figures. - how much data is available? Indicators vary in their data coverage across countries. The Business Environmental Risk Intelligence (BERI) rating covers approximately 50 countries primarily of interest to foreign investors who use the product. They also vary in their data coverage over time. Freedom House offers data from Others are much newer. These two considerations will determine how useful an indicator is for cross-country comparison or for measuring change over time. A third important consideration is whether the data on which the indicator is based, is up to date. Some surveys / figures are collected annually. Others are updated less frequently. - whether the indicator is specific or aggregate. Aggregate indicators are formed from multiple indicators. This results in more data being included in the preparation of ratings, which can enhance accuracy and country coverage (see below for discussion of the KKZ framework). They can be used to give a measure of a broad concept of governance (e.g. Rule of Law or Corruption). But the trade-off is that aggregate indicators are less clear about the precise location of problems and thus the reform needs or options. Indicators that are specific, look closely at a discrete aspect of governance but are less useful for giving a broad picture of the quality of governance (see discussion of 2 nd Generation Indicators). - Whether the indicator has a proven link to development or poverty outcomes. Numerous studies of the relationship between governance indicators and growth and development have been conducted. The results vary but overall there is agreement that high quality governance and public institutions are linked to growth in per capita income higher investment. For example, high civil liberties rating in the Freedom House index has 2

3 been linked to better rates of return in WB financed projects. Similarly, Collier and Dollar find that ratings for the World Bank s Country Policy and Institutional assessment (see below on CPIA) are positively linked to aid effectiveness. And Kaufman Kraay and Zoido- Lobaton argue that Governance matters because they find positive statistical relationships between high scores on rule of law, graft and voice and accountability and lower rates of infant mortality, higher literacy, and high per capita income. But the majority of these studies have been conducted utilising aggregate, subjectively based indicators, so establishing the relationship between poverty reduction and specific aspect of governance/ governance reform remains a challenge. Moreover, positive statistical relationships alone do not necessarily indicate a direct causal link. FIVE GOVERNANCE INDICATOR FRAMEWORKS 1. Kaufman, Kraay and Zoido-Lobaton aggregate indicators Summary - A comprehensive framework of governance indicators aggregated from 17 sources -most of which are subjective / perception based - Data available for 1997/8 and 2000/01 and for a high number of countries (175) relative to other frameworks - But large margins of error, and high degree of consistency between figures for 97/98 and 00/01 raises question about usefulness for precise measurement of change; - Complex method of aggregation of data results in low transparency Kaufmann Kraay and Zoido-Lobaton (KKZ) have constructed aggregate indexes from 14 different sources in 1998/98 (17 for the 2000/01 results). Annex 2 shows the sources used in the KKZ framework. The aggregate indexes are for 6 dimensions (clusters) of governance. Each of the 6 clusters contain indicators that are basically measuring the same aspect of governance, but which on their own are imperfect proxies. Box 2. Six clusters of Governance Indicators in the KKZ framework Cluster 1: Voice and accountability (process by which those in authority are selected and replaced) Cluster 2: Political Stability (likelihood of government being overthrown by violent means) 3

4 Cluster 3: Government Effectiveness (quality of public service provision, independence of civil service, government commitment to policies) Cluster 4: Regulatory quality (incidence of market unfriendly policies, etc) Cluster 5: Rule of Law (respect of citizens and the state for institutions which govern their interactions) Cluster 6: Control of Corruption Kaufmann et al have used their aggregate indicators to try to demonstrate the relationship between governance and three poverty outcomes: per capita income, infant mortality and adult literacy. They argue that countries scoring higher on the governance clusters tend to have lower mortality and higher literacy as well as higher per capita incomes. These results lead them to assert that there is a strong positive causal relationships from improved governance to better outcomes 3. A comparison between KKZ indicators in 1997/98 and 200/01 shows a high correlation across countries, reflecting persistence in perceptions of governance over this four-year period. Some countries show substantial change despite the large margin of error associated with the data. In addition it appears that changes in the different components move together on average. Zimbabwe for example showed an across-the-board deterioration 4. Whilst KKZ s analysis in 2002 suggested that significant change could be seen in some countries over time, they also emphasis that there is a high margin of error in their indicators given their origin in estimates/ perceptions. KKZ therefore suggest that users should be cautious in using their data to make comparisons across countries and over time. Instead, the aggregate indicators are more useful for identify(ing) broad groupings of countries with extremes of governance. 5 Two methodological characteristics of the KKZ framework aggregation and use of perception surveys- are important in the wider debate about governance indicators. KKZ forms its indexes by aggregating the figures from 14 (later 17) different sources. 3 Governance Matters (p.15), Kaufmann, Kraay and Zoido-Lobaton, Policy research Working Paper 2196, October 1999, World Bank 4 Governance Matters II; Updated Indicators for 2000/01 (p13) Kaufmann, Kraay and Zoido-Lobaton, Policy research Working Paper 2772, February 2002 World Bank 5 ibid p16. 4

5 Aggregation on one hand increases accuracy by introducing more data, and by balancing out idiosyncrasies in individual indexes. In addition, the aggregate provides a more precise signal of its corresponding broader governance concept than any of the component indicators would. Furthermore, it allows greater country coverage than would be possible if one source alone was considered in which case it is likely that there would be more countries missing: the KKZ data for 2001 covers 175 countries. But aggregation is problematic too, because the complex method by which data is aggregated reduces the transparency of the scoring process. And transparency and simplicity are highly desirable if developing country governments are to subscribe to indicators. The second characteristic of the KKZ framework is that most of the sources in the KKZ framework measure subjective perceptions of governance through surveys of experts and residents. Kaufman et al argue that such perception-based measures are valid (the findings correlate with other surveys and the fact that people/firms buy the surveys lends them credibility as sources). But surveys can also introduce the ideological bias of the organisations conducting them and may also produce results that are non-comparable because of questions being asked/answered in a culturally specific way. It is also worth noting that some of the surveys comprising the KKZ framework are only periodically updated, which makes them less useful for tracking change except over long time horizons. 2. World Bank / DAC Second Generation Indicators DFID has funded the Second Generation Indicators (2GI) Project within the World Bank s Poverty Reduction and Economic Management group (PREM) since July By contrast with the first generation of indicators that are largely subjective and aggregate indicators, this project aims to identify indicators that: a) are generated through a transparent process and therefore likely to be more politically acceptable b) are available across many countries and over time c) are high quality and accurate d) are operationally relevant as a result of being institutionally specific i.e. measuring a particular set of institutional arrangement within government or a defined output 6. 6 Specificity makes it possible to locate precisely which reforms are needed. With aggregate indicators the exact locus of the problems / reform needs are obscured. 5

6 e) use data that is available across many countries 7 and which can be periodically updated. f) are simple Applying the criteria above has resulted in the original set of eighty four candidate 2GI s being filtered down to around four which are emerging as the most promising 8. Some of the candidate indicators tested during the project are shown at annex 4. Candidate 2GI Description Data Contract intensive money Captures the proportion of money that is Regularly updated for countries held in banks as a measure of how much faith investors have in the government s ability /willingness to enforce financial contracts. included in the Financial Statistics Yearbook. Available for almost all countries over long periods of time. Policy (budgetary) volatility Treats budget as proxy for policy and measures median of year to year changes Data on expenditure by function is reported in IMF Government Finance in 14 functional classifications over 4 Statistics. Available for countries preceding years. Argues that coherence included in the GFS. and predictability of policy is important for businesses. Business Start-up procedures Measures obstacles faced by Currently available for 85 countries. entrepreneurs. Recognises that some No institutionalised process for level of regulation is efficient. updating data. Difficulty in pursuing Valid Complex and lengthy procedures for Data available for 109 countries. No Legal claims. pursuing claims through the court impede commercial activity. institutionalised process for updating data. The latter two of this group are arguably the most relevant to (DFID s) governance concerns. Business start-up procedures reflect the administrative capacity of the state, whilst the difficulty in pursuing valid legal claims gives a snapshot of the quality of the legal system. Systems for collecting data would need to be established if these two indicators were to be routinely used, however. Contract Intensive Money may be less relevant to the capacity of the state than to the quality of the financial sector. Policy Volatility is problematic as an 7 See towards more operationally relevant indicators of governance, PREM note 49, December 2000, World Bank. 2002s 8 See The set of possible 2GI s continues to grow and change. New ones being tested recently include a) control of the media: % owned by the government; b) level of transparency measured by timeliness with which government reports key statistics (using data that s already collected by the IMF) c) an indicator (scored on a scale of 1-7) of the competitiveness of legislative and executive elections, based on data contained in the WB s database of political institutions (Keefer et al) and d) the productivity of SOE s. 6

7 indicator since policy change can be either good or bad. This makes it difficult to attach meaning or value to changes observed. Another possible Second Generation Indicator is of legislative and executive electoral competitiveness using data from the World Bank sponsored Database of Political Institutions, described in full in section 5 below. The measures of electoral competitiveness in the DPI are transparently constructed (the coding is published) and they are specific. The project tested the importance of several 2GI s for 14 DAC poverty outcomes 9. Contract intensive money and policy volatility were included in the test. A strong positive correlation was found to exist between contract intensive money and the % of the population with access to safe drinking water. No significant relationships were found between policy volatility and any of the pro-poor outcomes used in the test. Recognising that a positive statistical relationship between say literacy and infant mortality is not enough to conclude a direct link 10, the second generation project has shifted to concentrate more on the relationships between 2GI s and intermediate variables or outcomes, such as investment. The results of this work are not available to date. The qualities of 2 nd generation indicators make them preferable in several ways to the subjective and less politically acceptable first generation indicators. Their main advantage is their greater operational relevance and their transparent compilation. But they have limits too. High specificity means that a 2GI alone cannot be relied upon to give a broad picture of governance. If a composite measure of governance is sought, it will be necessary to either aggregate 2GI s -thereby re-introducing problems of complexity and non-transparency, or to use them in parallel with other measures (as is proposed in this paper). Secondly, sole reliance on specific indicators may also create false incentives for governments to fix narrowly on specific reforms, but leave other problems untackled. Thirdly, 2GIs are possibly more demanding in terms of data gathering and updating than existing frameworks. Work on the 2GI project is still in progress. For example, strong emphasis has been placed on making them political acceptable and transparent through web-based discussion and the 9 The relationship between a few second Generation Indicators and Measure of Pro-poor Outcomes, Technical Working Note, Mark Kugler, July the 2 nd Generation project is cautious about claiming causal links between indicators and poverty outcomes, by contrast with the KKZ literature. 7

8 publication / dissemination of data. But it is clear that a broad based consensus has not yet emerged among experts and even less among developing country governments. 3. World Bank / IMF HIPC tracking: measuring the quality of PEM systems The HIPC public expenditure tracking survey (PETs) framework developed by IMF and World Bank is an attractive new indicator of the overall quality of a country s public expenditure management system. World Bank Country teams and the IMF s Fiscal Affairs Department assess countries against 15 benchmarks (as below) covering budget formulation, budget execution and reporting 11. Countries score from 1-15 depending on the number of benchmarks they meet. The first phase of the PETs exercise examined 25 HIPC countries, finding that 15 countries required substantial upgrading to their systems before poverty spending could be effectively tracked. Only 2 countries in the original set were deemed capable of tracking spending in a meaningful way. This framework has a number of strong merits. Firstly, effective public expenditure management has a clear link to poverty outcomes 12. Secondly, evidence suggests that this 11 See Actions to strengthen the Tracking of Poverty-Reducing Public Expenditure in Heavily Indebted Poor- Countries (HIPCs), March 2002, IMF/ WB. 12 What constitutes poverty-reducing expenditure, varies between countries. In the case of PETs is defined with reference to individual country PRSPs. 8

9 approach has been capable of picking up small changes in systems and practices, which other types of indicators (non-specific/aggregate indexes) would not be able to measure. Ghana s low score of 1 was thought to reflect changes in practice immediately following the change of government, though Bank staff felt that viewed over a 3-4 year period, Ghana should have compared more favourably to other countries in the first round of PET surveys. The operational usefulness of the PETs framework is also clear from the already common 9

10 use of results in the formulation of action plans on PEM. Thirdly, during the first phase of this exercise, significant amounts of consultation took place with governments, lending political acceptability to the framework. Finally, this is an objective framework in that it observes specific practices or features of systems. This objectivity also increases its potential acceptability with governments. The World Bank and IMF plan to roll-out this framework. PETS have already been carried out in non-hipc countries (Peru and the Dominican Republic) and over the next two years, fourteen African countries not in the original set of HIPC countries, will see surveys of this kind undertaken. In addition it has been agreed that data will be updated on an ongoing basis 13. Since there are no plans to alter the original methodology the 15 benchmarks will remain the same - time series data will soon be available, enabling change over time to be observed. Several issues need to be resolved if this indicator is to become more widely used, as is highly likely. The framework was developed specifically in relation to the HIPC initiative and has still been carried out in relatively few countries. Continuing tracking will require both the institutionalisation of data collection systems in countries already assessed and the creation of data collection systems in countries not included to date. Secondly, the assessments to date have been carried out by different teams (a combination of World Bank country teams and IMF Fiscal Affairs Department staff). As a result, there may be inconsistency between countries in the way that scores have been assigned to the 15 benchmarks. Work is underway to enhance consistency through the development of more rigorous guidance on the framework (e.g. what to look for in the case of the 15 benchmarks; technical advice on the differences between virtual poverty funds and institutional poverty funds etc.), but a small level of risk will remain since different teams will conduct the assessments. The last issue to highlight is the publication of data and results. Of the first generation of (HIPC) countries involved in the PETs exercise only 6 countries published the data (2 in Latin America, the rest in Africa). Governments inevitably will vary in their willingness to release this information. The WB and IMF also did not make the data widely available, though are now considering requesting that governments newly undertaking PETs work, agree to make the results public. Disclosure would reap gains in terms of transparency and 13 in summary at the end of 2003 and in a full update for the Bank and Fund boards in

11 accountability. However, the prospect of this data being released may provoke resistance in some countries (and possibly within WB and IMF if the results are seen in any way as a measure of the success of programmes). 4. Freedom House Summary - Long time series and annual production of data - One of few frameworks that deals with political issues - Subjectively compiled, increasing scope for controversy Freedom in the World is an annual survey of political rights and civil liberties compiled by Freedom House. Countries are assigned scores to numerical ratings from 1-7 (1 being the least free and 7 being the most free) against checklists of questions on political and civil rights 14. Annex 5 provides the full checklist of questions asked. These scores are then averaged to determine an overall status of Free (F) for average scores of 1-3, Partly Free (PF) for average scores of or Not Free (NF) for average scores of Survey teams -during the course of a year- draw on local and international news reports, NGOs publications, think tanks and academics as well as individual experts in preparing the ratings. Freedom in the World pre-dates many other sets of governance indicators, offering data on most countries on an annual basis from It is thus better established and widely known than other indexes. Being subjective and explicitly political in subject matter, Freedom House s index is probably the most contentious of the indexes and the most difficult for governments to accept (when they score badly). However, given the subject matter -civil and political freedomssubjective measurement is appropriate because it provides scope for nuances to be factored in.. 5. The database of political institutions Summary 14 See also for a full explanation of the methodology used for the annual survey Freedom in the World. 11

12 -Country data for 177 countries over 20 years -No normative value attached to one type of institutional arrangement over another -Strong relationship with other measures of democracy and political rights -Qualifies as a 2GI (is objective, specific and with data available over time and large number of countries) One alternative index to Freedom House, is the Database of Political Institutions (DPI), compiled by the World Bank and the Swiss Foreign Ministry 15. The DPI is an objectively compiled collection of data on political and institutional characteristics across 177 countries between Its significance is its large country coverage (all sovereign nations with populations above 100,000). DPI is also different to other politically focused databases. For example, the authors compare DPI with Polity III (Gurr, Jaggers and Moore 1998), which relies on subjective, and highly aggregated indicators. DPI by contrast claims to use variables that reflect only one dimension, so individual characteristics can be gleaned from the database to a much higher degree of specificity 16. The DPI gathers data on the seven variables. Annex 6 provides a full description of the variables and their interpretation. 7 variables in the Database of Political Institutions 1. Political System and Electoral competitiveness 2. Party preferences 3. Tenure turnover and popular vote share 4. Legislature 5. Electoral rules 6. Checks and balances 7. Federalism As part of the Second Generation Indicators project, the DPI s data on electoral competitiveness was compared with other indexes measuring democratic accountability and 15 See New Tools and new tests in comparative political economy: the Database of Political Institutions. Beck, Clarke, Groff, Keefer and Walsh, Development Research Group, WB. 16 The DPI also includes a wider range of variables than the database of political constraints (Henisz 1997), which focuses exclusively on checks and balances. 12

13 political rights 17. Significantly, strong similarities were found between the DPI s variable on legislative electoral competitiveness and a) Freedom House s score for political rights and b) the KKZ measure of voice and accountability, showing that DPI s measure of legislative competitiveness is consistent with other evaluations of accountability and political rights. But the tests also found that the DPI s indicator of legislative and executive electoral competitiveness was poor at measuring minor differences in degree between competitiveness in elected institutions. For example, 70% of legislatures scored 7 18, though it is highly likely that there are fine degrees of difference in competitiveness in these institutions. The possible advantage of the DPI measure of electoral competitiveness over Freedom House s measure of political rights is its objective compilation, and also the institutional specifity of what is measured, which is good for operational purposes. On the other hand, the DPI attaches no normative value to one institutional arrangement over another, so (except that DPI scores correlate with political rights) it is harder to claim that a certain score on the DPI is good or bad. Those values would need to be determined if the DPI index were put to operational use. CAPTURING GOVERNANCE IN PERFORMANCE BASED RESOURCE ALLOCATION The World Bank s Country Policy and Institutional Assessment Summary: - Rewards good performance and penalises bad performance on governance through application of a governance factor. - Is a subjective method of assessment based on the judgements of a large numbers of World Bank country staff. - Ratings are not disclosed, so limited potential for use outside the World Bank. 17 Legislative and Executive Indices of Electoral Competitiveness, Mark Kugler, World Bank (unpublished paper, 2002) 18 The DPI scores legislative electoral competitiveness on a scale from 1 to 7 where: 1 indicates that there is no legislature, 2 is for an unelected legislature, 3 is for an elected legislature with one candidate, 4 is for one party with multiple candidates, 5 is for multiple parties being legal but only one party wins seats, 6 is for multiple parties but largest party received 75%+ of seats and 7 is where largest party got less than 75% of seats. 13

14 The Country Policy and Institutional Assessment (CPIA) is the basic building block of the WB s performance based allocation (PBA) system for IDA resources. It is an annual performance assessment for all borrowing countries based on four groups of criteria: economic management, structural policies, policies for social inclusion and equity, and public sector management and institutions 19. The CPIA assessment criteria (20 in total) include six that concern governance explicitly. Box 1. Governance Items in CPIA framework B Economic Management 4. Management and sustainability of the Development Program D Public Sector Management and Institutions 16. Property Rights and Rule-based Governance 17. Quality of Budgetary and financial management 18. Efficiency of Revenue Mobilisation 19. Quality of Public Administration 20. Transparency, Accountability and Corruption in the Public Sector. Source: Linking IDA Support to Country Performance, April 2002 (p11) The Governance factor in the CPIA A high premium is placed on governance in the IDA allocation process, through calculation of a governance factor which rewards good performance on governance, and penalises poor governance 20. The methodology, which was changed after the 2001 Review of IDA 21, uses a graduated scale of penalties and benefits, rather the previous cut-off point below 19 The CPIA assessment (result weighted at 80%) is supplemented by the Annual Review of Portfolio Performance (ARRP), which assesses the quality of development projects and program management. The ARRP carries a weight of 20%. The CPIA and ARRP assessments are supplemented by the calculation of a governance discount (now called the governance factor). Together the CPIA, the ARRP and the Governance factor make up IDA s Country Performance Rating. 20 Additions to IDA Resources: Thirteenth Replenishment, Supporting Poverty Reduction Strategies, IDA November Page describes enhancements in the Performance Based Allocation System. 21 Review of the Performance Based Allocation System, IDA 10-12, February 2001, Operations Evaluation Department. The 2001 Review of IDA noted that the existing methodology had inequitable outcomes and only served to reduce IDA allocations to lowest ranking performers. A slightly higher rating allowed a country to escape the governance discount. A slightly lower rating could result in a large reduction in IDA allocations. For example, 7 of the countries that were subject to the governance discount in 1998 escaped it in 1999 (as a result of higher CPIA ratings). These ratings may have been warranted, but the fact that it took so little to achieve a doubling in IDA allocations suggested that the system was not very robust. 14

15 which IDA allocations were markedly reduced. The Governance factor is calculated by dividing the average governance rating by 3.5, and applying an exponent of 1.5 to this ratio. Governance factor = (governance rating /3.5) 1.5 A country s overall CPIA rating is multiplied by the governance factor, resulting in an increase or a decrease in the overall rating, depending on the degree to which the country s governance rating is satisfactory (above 3.5) or unsatisfactory (below 3.5). In this way a country that scores above the mid-point on the governance-related criteria will receive a premium and a country that scores below the mid-point will receive a discount. Scoring process in the CPIA World Bank country teams carry out the CPIA. They use a questionnaire defining 20 criteria (including the 6 governance items quoted above) and what would constitute good and unsatisfactory ratings (1 = unsatisfactory for three or more years, 6 = good for three or more years). In some cases the questionnaire refers staff to certain established datasets / databases for some of the 20 criteria 22. Thus the final CPIA scores constitute the judgements of a number of individual staff on a mix of objective data (e.g. provided by databases) and of more subjective assessments of how satisfactory x country is against a range of criteria. There has been an ongoing debate about disclosure of CPIA scores. Full disclosure is resisted because of the possible repercussions in markets and in reducing levels of country openness. But following the IDA review, there has been a move to disclosing country results by quintile (though not individuals country ratings). The prohibition on disclosure of CPIA ratings limits the usefulness of the framework outside the Bank. However, the approach could be replicated both in content and in scoring approach in other agencies 22 For example, on Efficiency of Revenue Mobilisation (item 18) World Development Indicators table 5.5 on Tax Policies is the guidepost; and on Quality of Public Administration (item 19) the World Bank s database on Civil Service Wages and employment 22 along with other sources of guidance, are suggested. For the remaining Governance items (Management and Sustainability of the Development Program (4), 16. Property Rights and Rule-based Governance (16), Quality of Budgetary and financial management (17), Transparency, Accountability and Corruption in the Public Sector (20)) the guidance tells staff what to look for and offers descriptions of scenarios against each score from

16 where there was an interest in capturing the quality of governance in performance based allocation systems There were proposals within the Bank to complement CPIA ratings with external datasets to increase objectivity and accuracy, though these never gained acceptance. 16

17 Conclusion: Which indicators? The preceding discussion explains the trade-offs associated with using one or other set of governance indicators. The key trade-offs are between subjective and objective indicators and between aggregate and specific indicators. Different indicators frameworks will be useful for different purposes. Governance indicators for use at country level for projects need not have the same characteristics as an indicator needed for making comparisons across a large range of countries. The former could for example, rely on a data collection system that is present in country x but not in other contexts. An indicator that will serve as the basis for a partnership between countries (e.g. a peer review process) needs to be acceptable to the countries involved and be simple enough for data collection and updating to be managed where capacity is low. Indicators to be used as a basis for identifying interventions and strategies need to operationally relevant or institutionally specific. Institutionally specific indicators alone, on the other hand, are probably of little value for aid allocation processes where a broad picture of multiple aspects of governance needs to be captured. Annexes 7,8 and 9 illustrate how countries fair in the various frameworks included in this paper. Comparing the ratings across a random sample of Low Income countries (annex 7) and comparing the worst scoring countries in 4 frameworks (Vietnam and Rwanda feature in ¾ groups, Cameroon as a low scorer in both the HIPC framework and in TI s corruption index) serves primarily to highlight varying levels of country coverage in the different indicator frameworks. KKZ covers all of the Low Income countries sampled, whereas HIPC, TI and the 2GI indicators cover far fewer. A comparison of four selected countries (Bolivia, Indonesia, Uganda and Vietnam) in the different frameworks ranks Bolivia highest (see annex 9). Simple comparison also suggests a degree of consistency between the frameworks: both Freedom House and KKZ give the countries the same rankings on political freedom and voice and accountability respectively. Bolivia also ranks worst on both the 2GI indicator of business start-up procedures and KKZ s index of government effectiveness. There is no similarity in the rankings for these countries in the index on Rule of Law (KKZ) and the 2GI on pursuing valid legal claims. 17

18 In conclusion this paper offers a set of governance indicators, which if used in tandem, give a broad picture of governance and combine the full range of methodological features of indicators. The assumptions behind the choice of this set are a) that it is desirable to use some indicators that developing country governments will subscribe to as a basis for partnership and for dialogue about specific reform measures. b) that aggregation is necessary in some instances to increase data coverage and to provide a broad picture (e.g. on corruption), but should not be relied on solely, because aggregation reduces transparency in the preparation of ratings c) that it is valuable to use a mixture of subjective and objective indicators. Democracy for example lends itself to subjective assessment and d) that whilst ideally we would want to use data sets that already exist, if good new indicators are found, it is worthwhile investing in new data collection systems. A possible set of indicators - Business start-up procedures (2GI). This indicator is useful because of its relevance to investment. It may also indicate the quality of administration and likely levels of graft. Its strength is that it is objective, specific and transparently constructed. Its weakness is that though data is available for 85 countries, data collection systems for this indicator have not been institutionalised. - Difficulty in pursuing valid legal claims (2GI). This gives a picture of the quality of the legal system, which has a bearing on investment, but may also be relevant to wider questions of access to justice. As with business start up procedures, this indicator is objective, transparent and specific. - World Bank / IMF (HIPC) public expenditure tracking surveys. This is the most robust index of the quality of PEM, and is both objective and specific about areas for improvement. Though operationalised in less than 30 countries to date, this framework is soon to be rolled out further. It tells us about government effectiveness and accountability and is important for donors providing direct budget support. - Freedom House Indicators of political rights and civil liberties (Freedom in the World). This is a well-established framework describing trends in democracy, for which data is available annually over a wide range of countries. Its subjective nature might attract controversy, but is also defensible given the subject matter. 18

19 - KKZ s aggregate indicator of control of corruption. As with the other aggregate indicators in the KKZ framework, this measure combines the ratings from a number of different surveys of experts and businesses- on perceptions of corruption. Aggregation makes this a complex indicator to replicate and understand, but it also reduces the idiosyncratic measurements error in each source and increases country coverage. Kathryn Casson July

20 Annex 1: Selected Performance Indicators and their characteristics Sources WDR97 (Private sector survey) Aspects of Governance Assessed (partial list only for some sources) Policy unpredictability, Quality of government services, Corruption and red tape, and Judicial unpredictability Specificity Med CPIA (World Bank) Property Rights and Rule-Based Med Governance Quality of Budgetary & Financial Management Efficiency of Revenue Mobilization Efficiency of Public Expenditures Transparency, Accountability and Corruption Method of Data Collection Business Survey Experts (many) Coverage across countries Coverage over time Reliance on Subscribers (y/n) Med Low No Low High Low No None Use in Published Studies KKZ (Kaufmann, Kraay and Zoido- Lobaton) Graft Rule of Law Voice and Accountability Political Instability and violence Government Effectiveness Regulatory Burden Low Aggregation High Low No Low TI (Transparency International) Corruption Perceptions Index Low Aggregation Med Low No Med Freedom House Political Freedoms Civil Liberties Low Experts (few) High High No High International Country Risk Guide (ICRG) Corruption in Government Law and Order Tradition Bureaucratic Quality Low Experts (few) High High Yes High BERI (Business Environmental Risk Intelligence) Bureaucratic Delays Contract Enforceability Nationalization Risk Policy Stability Low Experts (many) Low High Yes Med Heritage Property Rights Black Market Regulation Low Experts (few) High Low No Low GCR (Global Competitiveness Report WCY (World Competitiveness Yearbook) CIM (Contractintensive Money Civil service independence from Med politics Competence of public sector personnel Tax evasion Effectiveness of police force Bribing and corruption Med Tax evasion Public service exposed to political interference Personal security and private property Contract enforcement and property rights Business Survey Business Survey Low Low No Low Low Low No Low Med Objective High High No Low Policy volatility Data[xls 23 K] Telephone delays Policy credibility and fiscal management Quality of government service delivery High Objective Med Med No None Med Objective High Med No Low Source: 20

21 Aggregate Governance Indicators Dataset: The composite governance indicators are based on data from selected variables provided by the following sources: Source Name: Internet Address: Publication: Business Environment Risk Intelligence Business Risk Service Columbia University State Capacity Project Economist intelligence Unit Country Risk Service European Bank for Reconstruction and Development Transition Report Freedom House Nations in Transition Freedom House Freedom in the World Gallup International Gallup Millenium Survey Institute for Management Development World Competitiveness Yearbook Latinobarometro Latinobarometro Surveys Political Economic Risk Consultancy Asia Intelligence Political Risk Services International Country Risk Guide PriceWaterhouseCoopers Opacity Index Standard and Poor's DRI/McGraw-Hill Country Risk Review World Bank Business Enterprise Environmnet Survey World Bank World business Environment Survey World Economic Forum Global Competitiveness Survey/Africa Coverage: 50 mostly developed countries 109 developed and developing countries 115 developed and developing countries 26 transition economies 27 transition economies 192 developed and developing countries 60 mostly developed countries 49 mainly developed countries 17 Latin American countries 14 Asian countries 140 developed and developing countries 35 developed and developing countries 111 developed and developing countries 18 transition economies 81 developed and developing countries 75 developed and developing countries Annex 2: Data Sources in the KKZ aggregate indicators framework 21

22 Annex 3: Four country comparison using KKZ data Voice and Accountability Political Stability/No Violence Country BOLIVIA INDONESIA UGANDA VIETNAM Income Category Lower Middle Income Low Income Low Income Low Income Dataset Percentile Income Percentile Rank World Category Rank (0-100%) Average Average (0-100%) World Average Income Category Average 2000/ / / / / / / / Government Effectiveness Regulatory Quality Country BOLIVIA INDONESIA UGANDA VIETNAM Income Category Lower Middle Income Low Income Low Income Low Income Percentile Income Percentile Income Rank World Category Rank World Category Dataset (0-100%) Average Average (0-100%) Average Average 2000/ / / / / / / / Rule of Law Control of Corruption Country BOLIVIA INDONESIA UGANDA VIETNAM Income Category Lower Middle Income Low Income Low Income Low Income Percentile Income Percentile Income Rank World Category Rank World Category Dataset (0-100%) Average Average (0-100%) Average Average 2000/ / / / / / / /

23 Annex 4: Second Generation Indicator candidates and their key characteristics Indicator/Characteristics Description Link to Poverty/ development outcomes 1. Public Administration.- Trade Tax Reliance on revenue from trade taxes? -TT revenue reflect other influences Revenue reflects weak administrative capacity -Can depend on size of country 2. Performance and Outputs Contract Intensive Money 3. Performance and Outputs- Budgetary / Policy Volatility Proportion of money held in bank accounts, indicating how much faith in the gov s ability to enforce financial contracts and refrain from expropriation. Volatile and unpredictable government reduces private investment Statistically significant association with 8/14 of DAC Outcomes used in test by 2 nd Generation project. Statistically significant association with 5/14 of DAC Outcomes used in test by 2 nd Generation project. Accuracy/Quality Data availability Transparency/Political acceptability -there are other ways of holding currency ( gold or foreign currency) for which there is limited data -policy changes can sometimes be from inefficient to efficient allocations -Data regularly updated for countries included in Governance Financial Statistics -as % of gov. revenue, available in WB WDI s. -available, but not for alternatives ( see accuracy/ quality) -regularly updated for countries included in GFS -good, since revenues are reported to IMF adjusting for other influences would reduce transparency. -data used to measure contract intensive money all reported to IMF and included in the Fin. Stats yearbook. -expenditures by function, that are used to calculate this are reported to the IMF and are in GFS 4. Accountability Institutions- delays in Auditing 5. Civil Service- Average Wage to Financial Sector Wage -Elapsed time between end of FY and tabling of externally audited financial statements. Compares government wages to financial sector wages 6. Weight of Public Employment -Tests whether gov. employment levels are appropriate to country s population. Measure as % of population in the civilian central government. 7. Fiscal Decentralisation -Devolution can help improve services and accountability. This indicator has 6 sub-components: subnational share of expenditure, subnational share of revenues, subnational expenditures as share of GDP, tax share, vertical imbalance. 8. Surveys of Business Persons (Objective) 9. Surveys of Business Persons (perceptions of Governance) 10. Difficulty in pursuing valid legal claims -To evaluate characteristics of the business environment -To evaluate characteristics of the business environment -Where procedures are complex and lengthy, commercial activity suffers. Indicator based on data on actions? -timeliness is important but other aspects of audit performance get left out here (eg independence of audit institutions). No statistically significant -non monetary benefits are not association found with 14 of DAC s poverty outcomes. captured Statistically significant association with 5/14 of DAC Outcomes used in test by 2 nd Generation project. -financial sector employees tend to be on the high end of white collar employment, so this measure needs to be considered in conjunction with other measures of gov wages and competitiveness, Civilian Central Government is only one category of total gov. employment. So needs to be considered in conjunction with other measures. -each sub indicator needs to be considered alongside the others all sub parts only relate to fiscal aspects 23 -no systematic crosscountry data has yet been conducted -data collected on a country by country basis so comparability is difficult. -Available on Bank s civil service wage database for numerous countries -collected on country by country basis, so comparability problem. Available on Bank s civil service wage database for numerous countries -Available through WB s fiscal decentralisation indicators databased for numerous countries.?? -constructed from WDR 97 surveys?? -constructed from WDR 97 surveys -plus some in ( Business Environment and Enterprise Performance? -data reflects only simple cases where facts are not in dispute, and where claims are small. Survey (BEEP s) -data available for 109 countries. -adjusting for other effects of volatility would increase accuracy but reduce transparency -can be calculated from information provided by govs. -calculated using secondary data such as WB Public Expenditure reviews. Av. Fin Sector wage uses ILO methodology -calculated using secondary data such as WB Public Expenditure reviews. -calculated using the IMF s Gov., Financial Statistics handbook. -because asking for factual information, high on transparency and political acceptability? -data collection process published and could therefore be challenged.

24 required to file a complaint and to obtain and enforce a judgement in 109 countries. 11. Business Start up procedures -Based on data about number of procedures, costs of fees, number of business days etc, required before company can operate legally. 12. literacy Rate? -data reflects only officials requirements ( ie not bribes) -data available for 85 countries, but no institutionalised process for updating the data. -data collection process published and could therefore be challenged. 24

25 Annex 5 : Freedom House methodology for annual survey of Freedom in the World The survey rates political rights and civil liberties separately on a scale of 1 to 7, with 1 representing the most free and 7 the least free. A country or territory is assigned to a particular rating based on the individual survey authors responses to a series of checklist questions and the judgments of the survey team at Freedom House. The authors assign initial ratings to countries or territories by awarding from 0 to 4 raw points per checklist item, depending on the comparative rights or liberties present. (In the surveys completed from through , the methodology allowed for a less nuanced range of 0 to 2 raw points per question.) The only exception to the addition of 0 to 4 raw points per checklist item is additional discretionary question B in the political rights checklist, for which 1 to 4 raw points are subtracted depending on the severity of the situation. The highest possible score for political rights is 32 points, based on up to 4 points for each of eight questions. The highest possible score for civil liberties is 56 points, based on up to 4 points for each of fourteen questions. After the countries and territories have been assigned political rights and civil liberties ratings based on the total number of raw points in each of the two categories, the survey team makes minor adjustments to account for factors such as extreme violence, the intensity of which may not be reflected in answering the checklist questions. Political Rights Check List Is the head of state and/or head of government or other chief authority elected through free and fair elections? Are the legislative representatives elected through free and fair elections? Are there fair electoral laws, equal campaigning opportunities, fair polling, and honest tabulation of ballots? Are the voters able to endow their freely elected representatives with real power? Do the people have the right to organize in different political parties or other competitive political groupings of their choice, and is the system open to the rise and fall of these competing parties or groupings? Is there a significant opposition vote, de facto opposition power, and a realistic possibility for the opposition to increase its support or gain power through elections? Are the people free from domination by the military, foreign powers, totalitarian parties, religious hierarchies, economic oligarchies, or any other powerful group? Do cultural, ethnic, religious, and other minority groups have reasonable selfdetermination, self-government, autonomy, or participation through informal consensus in the decision-making process? Category Number Raw Points Civil Liberties Check List A. Freedom of Expression and Belief 1. Are there free and independent media and other forms of cultural expression? (Note: in 25