2006 Global Integrity Country Assessments and Global Integrity Index: Methodology White Paper

Transcription

1 2006 Global Integrity Country Assessments and Global Integrity Index: Methodology White Paper The Global Integrity Approach Global Integrity's Integrity Indicators provide the underpinning for our core reporting and analysis on governance and corruption. The Indicators represent the world's most comprehensive data set providing quantitative data and analysis of corruption, rule-oflaw, openness, and business climate in diverse countries around the globe. Utilizing our network of several hundred on-the-ground analysts and our unique scoring system, Global Integrity generates actionable data and qualitative analysis from the indicators that assess a country's openness and potential for corruption. The Integrity Indicators are based on a simple yet revolutionary concept. Rather than trying to measure actual corruption, considered virtually impossible by experts, Global Integrity quantitatively measures the opposite of corruption, that is, the access that citizens and businesses have to a country's government, their ability to monitor its behavior, and their ability to seek redress and advocate for improved governance. The Integrity Indicators break down that "access" into a number of categories and questions, ranging from inquiries into electoral practices and media freedom to budget transparency and conflicts of interests regulations. We unpack these concepts by looking not only at what laws or institutions are on the books but by assessing their staffing levels, budget independence, political independence, and citizen access to the most important anticorruption mechanisms. Combined with vivid narration prepared by our unparalleled team of leading journalists, the extensive data provided by the Integrity Indicators informs and empowers citizens, activists, donors, businesses, and governments in any country. Global Integrity provides detailed data and comparative country-by-country assessments of the mechanisms in place to prevent abuses of power and promote public integrity. Using a blend of social science and journalism, in-country teams of independent social scientists and investigative journalists report on the de jure as well as de facto reality of corruption and anti-corruption mechanisms. While the Reporter s Notebook on the culture of corruption often paints a depressing picture (how corruption looks, tastes, feels and smells to the average citizen), the Integrity Indicators identify strengths and weaknesses in the national anti-corruption architecture and serve as a road map for possible reforms. Transparency, both in terms of our methodology and findings, is what characterizes the Global Integrity approach and enhances the robustness and credibility of our findings. As we continue to improve our methodology and learn from our field work experiences, we welcome your critical feedback Global Integrity th Street N.W. Suite 1040 Washington, D.C USA Website: info@globalintegrity.org Office: Fax:

2 Washington Team The core Global Integrity team in Washington [hyperlink to Who We Are staff page on site] both identifies and manages our teams of local, in country reporters and researchers who are primarily responsible for collecting the raw reporting and data that ultimately generates the Global Integrity country assessments and Global Integrity Index. The core staff in Washington undertakes additional research and analysis for the various components of the annual Global Integrity Report and other projects, as well as ongoing fundraising, communications and outreach activities. In 2006, Global Integrity benefited greatly from the feedback of an informal methodology advisory committee set up to provide technical assistance and feedback on methodological and scoring issues. The group included the following experts acting in their personal capacity: Omar Azfar (IRIS) Monica Dorhoi (World Bank) Gregory Kisunko (World Bank) Andrew Greene (USAID) Jennifer Gauck (ABA-CEELI) Leslie Harper (Inter-American Development Bank) Gregory Kisunko (World Bank) Steve Knack (World Bank) Gerardo Munck (University of Southern California) Brad Parks (Millennium Challenge Corporation) Gary Reid (World Bank) Joe Siegle (DAI) Fernando Straface (Inter-American Development Bank) We are grateful for these individuals feedback and encouragement. Global Integrity is responsible for all interpretation and assessments of the data and reporting as well as for any mistakes made. Fieldwork and Country Selection From June to December 2006, Global Integrity conducted field research (strictly covering the period June 2005 to end of May 2006) in the following 43 countries or territories: Argentina, Armenia, Azerbaijan, Benin, Brazil, Bulgaria, Cambodia, Democratic Republic of Congo, Egypt, Ethiopia, Georgia, Ghana, Guatemala, India, Indonesia, Israel, Kenya, Kyrgyz Republic, Lebanon, Liberia, Mexico, Montenegro, Mozambique, Nepal, Nicaragua, Nigeria, Pakistan, Philippines, Romania, Russia, Senegal, Serbia, Sierra Leone, South Africa, Sudan, Tajikistan, Tanzania, Uganda, the United States of America, Viet Nam, the West Bank, Yemen, and Zimbabwe. 2

3 While our ambition remains to cover all countries and regions of the world, the final set of countries for 2006 was chosen according to the following criteria: 1. Balance: Global Integrity seeks to ensure a basic geographic balance in its annual rounds of country assessments. For 2006, the breakdown is as follows: Africa: 15 (Benin, Democratic Republic of Congo, Ethiopia, Ghana, Kenya, Liberia, Mexico, Mozambique, Nigeria, Senegal, Sierra Leone, South Africa, Sudan, Tanzania, Uganda, and Zimbabwe) South East Asia: 4 (Cambodia, Indonesia, Philippines, Viet Nam) Europe: 5 (Bulgaria, Montenegro, Romania, Russia, Serbia) Latin America: 5 (Argentina, Brazil, Guatemala, Mexico, Nicaragua Middle East: 5 (Egypt, Israel, Lebanon, the West Bank, Yemen). South and Central Asia: 8 (Armenia, Azerbaijan, Georgia, India, Kyrgyz Republic, Nepal, Pakistan, Tajikistan) North America: 1 (United States of America) In selecting countries we referred to Freedom House s annual Freedom in the World survey [hyperlink to to ensure balanced coverage. Of this year s countries, 15 were ranked by Freedom House in 2006 as Free, 17 were Partly Free, and 11 were Not Free. 2. Budget: The current budget of Global Integrity limits our ability to undertake an increasing number of country assessments although our goal remains to expand our coverage to true global coverage, at least on an alternating year basis. The total number of countries covered in 2006, 43, was the maximum number of countries we could afford to assess thanks to the generous support of our current funders [link to Support Us page]. 3. Availability of experts: Perhaps the most important criteria that affects whether a country is selected is whether Global Integrity is able to recruit a sufficiently qualified research team in the country. We cannot, and do not, carry out country assessments where we lack confidence in our team. See additional details below on how country teams are recruited and selected. 4. Emphasis on large aid recipient countries: For 2006, our financial support from the World Bank included assessing a number of International Development Association [hyperlink to (the classification for the Bank s very poorest client countries). This explains the larger selection of sub-saharan African countries compared to other regions. 3

4 5. Emphasis on emerging markets: For 2006, Global Integrity also sought to increase its coverage of emerging markets in order to appeal to the private investment community. As such, our 2006 list of countries covered includes eleven of the 25 countries defined as emerging markets according to Morgan Stanley Capital International s (MSCI) July 2006 Emerging Markets Index [hyperlink to 6. Appeal to the international policy community: For 2006, we included 14 countries that were either eligible or threshold countries under the Millennium Challenge Corporation s [hyperlink to criteria. Thirty eight of our 2006 countries are signatories to the U.N. Convention Against Corruption [hyperlink to Our 2006 sample is by no means globally representative but provides interesting country coverage across several key variables. Country Teams Global Integrity combines the skills of leading journalists with expert social scientists and researchers to produce its comprehensive country assessments. In each country, Global Integrity retained a team of at least four or five researchers, working independently, to compile the material for the country assessments. A Global Integrity country team typically consists of: A lead reporter who writes the Reporter s Notebook. A lead social scientist who compiles the initial scores for the Integrity Indicators. Two to six additional country readers (a mix of other in-country experts and out-of-country experts) who blindly review the raw data and reporting to provide additional perspectives and serve as part of the quality control mechanism. The teams are coordinated from Washington via the Internet and phone. Until the public release of the country assessments, the social scientists, journalists, and peer reviewers are unaware of the identities of other members of the country team. This is done to maintain the independence of the individual responses and avoid a peer-influenced consensus. Global Integrity s country assessments are produced through a coordinated effort of a large number of researchers and journalists, the vast majority of them located in-country. To identify appropriate social scientists, journalists, and peer reviewers in each country, Global Integrity actively recruits potential team members through informal partnerships with several well-placed international NGOs that work in the anti-corruption and good governance field. We also heavily recruit at international conferences and seek referrals 4

5 from colleagues with whom we already work in various countries. Prospective team members undergo a thorough background check to verify their expertise and independence (as a policy, Global Integrity does not retain an individual if the individual works in or for the government). For 2006, we received around 550 CVs from individuals who expressed interest in serving on a country team in one of our three distinct capacities (journalist, lead researcher, or country reader). After a competitive internal selection process, the most qualified and available social scientists and journalists in each country were identified, invited, and sent a contract with specific instructions on the scope of the work. The total number of country team members for 2006 was more than 220. The Country Reports For each of the 43 countries, Global Integrity produces an online Country Report that includes: Country Facts: Prepared by Global Integrity staff, these are drawn from a variety of publicly-available sources and provide basic political and economic background information on every country assessed. Corruption Timelines: compiled by Global Integrity researchers in Washington, these are a unique political/historical timeline of significant corruption-related events at the national level. Designed as a quick reference resource, each timeline summarizes the main corruption-related events in the country from the early 1990s through present-day. Based largely on reliable English-language international and national media sources, the Corruption Timeline pulls together in an easy-to-read, chronological fashion what is essentially in the public domain about corruption in a particular country. The Reporter s Notebooks: prepared by the lead in-country journalist, these are 1,250-word original essays by leading journalists on the culture of corruption and state of governance in a particular country. This hard-hitting, impressionistic essay provides a snapshot of corruption in day-to-day life as well as the recent history and context regarding the most high-profile corruption cases dominating that country s media. The Integrity Scorecard: prepared by the lead in-country social scientist/researchers. Each country s Integrity Indicators are aggregated to generate the Global Integrity Index (see additional details below). Details on the Integrity Scorecard The Integrity Scorecard for each country examines three things: 1. The existence of public integrity mechanisms, including laws and institutions, which promote public accountability and limit corruption. 5

6 2. The effectiveness of these mechanisms. 3. The access that citizens have to these mechanisms. More specifically, indicators of existence assess the laws, regulations, and agencies/entities or equivalently functioning mechanisms that are in place in a particular country. Indicators of effectiveness assess such aspects of public integrity as protection from political interference; appointments that support the independence of an agency; professional, full-time staff and funding; independently initiated investigations; and imposition of penalties. Indicators of citizen access assess the ready availability of public reports to citizens, or publicly available information, within a reasonable time period and at a reasonable cost. The Integrity Indicators are a unique instrument designed to provide a quantitative assessment of anti-corruption safeguards in a particular country. Carefully selected from a comprehensive review of the anti-corruption literature and other democratic governance sources, and building on Transparency International s National Integrity Systems approach to assessing governance, the Integrity Indicators are used to score the institutional framework that exists at the national level to promote public integrity and accountability and prevent abuses of power. For 2006, the 75 core Integrity Indicators (290 including sub-indicators) were organized into six main governance categories and 23 sub-categories. They are: I Civil Society, Public Information and Media I-1 Civil Society Organizations I-2 Media I-3 Public Access to Information II II-1 II-2 II-3 III III-1 III-2 III-3 III-4 IV IV-1 IV-2 IV-3 IV-4 Elections Voting & Citizen Participation Election Integrity Political Financing Government Accountability Executive Accountability Legislative Accountability Judicial Accountability Budget Processes Administration and Civil Service Civil Service Regulations Whistle-blowing Measures Procurement Privatization V Oversight and Regulatory Mechanisms V-1 National Ombudsman V-2 Supreme Audit Institution V-3 Taxes and Customs 6

7 V-4 Financial Sector Regulation V-5 Business Licensing and Regulation VI VI-1 VI-2 VI-3 VI-4 Anti-Corruption Mechanisms and Rule of Law Anti-Corruption Law Anti-Corruption Agency Rule of Law Law Enforcement Click here [hyperlink to Excel file provided by Global Integrity] to download the 2006 Integrity Indicators. Generating an Integrity Scorecard Each Integrity Indicator is scored directly by the lead social scientist and substantiated as far as possible with relevant references and additional comments. The data is relayed from the field to HQ via the internet using a password-protected web site. There are two general types of indicators: in law and in practice. All indicators, regardless of type, are scored on the same ordinal scale of 0 to 100 with zero being the worst possible score and 100 perfect. In law indicators provide an objective assessment of whether certain legal codes, fundamental rights, government institutions, and regulations exist. These de jure indicators are scored with a simple yes or no with yes receiving a 100 score and no receiving a zero. In practice indicators address de facto issues such as implementation, effectiveness enforcement, and citizen access. As these usually require a more informed and subjective assessment, these in practice indicators are scored along an ordinal scale of zero to 100 with possible scores at 0, 25, 50, 75 and 100. Lead researchers are required to provide a reference to substantiate each of their scores. This may be an interview conducted with a knowledgeable individual, a website link to a relevant report, or the name of a specific law or institution, depending on the particular indicator. As an incredibly rich research tool we plan to expand and harmonize the types of references for each of the indicators in future iterations of the index. Lead researchers are also offered the opportunity to include additional comments to support their score and reference for a particular indicator. These are particularly useful in capturing the nuances of a particular situation, namely the Yes, but phenomenon which is often the reality in undertaking this type of research. Personality, language, and culture can all affect the interpretation of a particular indicator and the score assigned to it. To minimize this effect and maximize inter-coder reliability, Global Integrity, in consultation with the methodology advisory committee, developed a master codebook for scoring the 2006 Integrity Indicators. The Codebook anchors every single indicator and sub-indicator to a predefined set of criteria. In essence, the scoring 7

8 criteria guide the lead researcher by suggesting, If you see X on the ground, score this indicator in the following way. For binary yes/no in law indicators, scoring criteria are provided for both yes (100) and no (0) responses. For in practice indicators, scoring criteria are defined for each of the 100, 50, and 0 scores with 25 and 75 deliberately left undefined to serve as in between scoring options. The master Codebook is downloadable here [hyperlink to Excel file to be provided by Global Integrity]. In summary, a given indicator or sub-indicator has the following elements: Indicator question, provided by Global Integrity Indicator scoring criteria/coding, provided by Global Integrity Indicator score (either yes (100)/no (0) or ordinal scale of with steps at 25, 50, and 75), assigned by the lead researcher based on o References, provided by the lead researcher o Comments (optional), provided by the lead researcher o Reader review comments (optional), as provided through a blind peer review process (See more on the peer review process below.) Example of an Integrity Scorecard: A powerful tool for analysis The data presented in an Integrity Scorecard is a powerful and unique way to succinctly assess the strengths and weaknesses of a country s national integrity framework (i.e. the existence and effectiveness of the mechanisms in place to prevent abuses of power). The data can be used to prioritize governance challenges and allocate available resources in a more efficient manner, for instance to strengthen the independence of the judiciary, or tighten up transparency around political financing. Integrity Scorecards are also a valuable tool for grassroots advocates seeking to target their advocacy programs on key priority sectors. Generating the Global Integrity Index: A Tool for Cross-Country Analysis While it is important to stress that the strength of the Integrity Indicators is the Integrity Scorecard they generate for an individual country (which provides an in-depth diagnostic tool for users), Global Integrity is also able to generate scores across categories and countries and classify countries into an overall Global Integrity Index according to various performance bands. These may be useful tools for those wishing to compare countries overall performances against each another. The Global Integrity Index groups countries into five performance tiers generated from the scores assigned to the individual integrity indicators: Very strong (90-100) Strong (80-90) Moderate (70-80) Weak (60-70) Very Weak (Below 60) 8

9 For the purpose of producing the Global Integrity Index, a simple aggregation method is used that parallels the process for generating indicator, sub-category, and category scores for each country. Original indicator and sub-indicator values are assigned by the lead social scientist for the country (those scores are adjusted following the peer review process; see more below). Each indicator score is then averaged within its parent subcategory, which produces a subcategory score. The subcategory score is in turn averaged with the other subcategory scores in a parent category. Category scores are averaged to produce a country score which then falls into one of the above five groupings. Because some aspects of governance integrity are harder to measure definitively, some categories require a more complex matrix of questions than others. Thus, the categories are equally valued, even if some categories are derived from a more lengthy series of questions than others. Similarly, the subcategories are equally valued within their parent category. The same comparison cannot be made between subcategories in different parent categories. In other words, each score (sub-indicators, indicator, and so on) is equally weighted with its peers addressing the same topic. However, indicators from different categories are not equally weighted. Our approach of using equally valued concepts and adding subordinate elements as needed has produced score weightings that reflect the six main conceptual categories. Peer Review Process The importance of the peer review process cannot be overstated. Since Global Integrity utilizes an expert assessment approach to compile the Integrity Indicators and Reporter s Notebook, it is crucial that we employ quality control mechanisms to ensure that our data and reporting is as accurate and as balanced as possible. Individually contracted and carefully vetted readers, selected for their independence and expertise in particular countries, are asked to blindly review both the raw Reporter s Notebook and the raw Integrity Indicators using the same password protected website used to file the original data and reporting from the field. The blind review process ensures that readers are unrestrained in their commentary, which most likely adds to 9

10 frankness when commenting on the draft country report. Reader comments are used to interpret and in some cases adjust scores that reviewers identify as containing errors, bias, or out-of-date information. Score adjustments follow certain rules and generally require repetition (i.e. similar comments from several reviewers) or solid referencing of a factual dispute. See further details on this below (Final Index Scores). In reviewing the Reporter s Notebook for the country, readers are asked to consider the following: Is the Reporter s Notebook factually accurate? Is the Reporter s Notebook fair? Is anything misrepresented or unclear? Are there any significant events or developments that were not addressed? Reader comments on the Reporter s Notebook are captured in narrative, paragraph style and are published anonymously alongside the final Reporter s Notebook. For the Integrity Indicators, readers are asked to consider the following: Is the particular Indicator or Sub-indicator scored by the lead researcher factually accurate? Are there any significant events or developments that were not addressed? Does the Indicator or Sub-indicator offer a fair and balanced view of the anticorruption environment? Is the scoring consistent within the entire set or sub-set of Integrity Indicators? Is the scoring controversial or widely accepted? Is controversial scoring sufficiently sourced? Are the sources used reliable and reputable? The reader process for the data scorecard, like for that of the Reporter s Notebook, does not assign direct attribution of reader comments. This ensures that readers are unrestrained in their commentary. Reader comments on the country s data scorecard are included alongside the final scorecard and play an important role in final scoring adjustments prior to publication. For 2006, Global Integrity retained the services of approximately 130 readers for the 43 country assessments, with some readers reviewing a number of countries [hyperlink to the part of Who We Are page that lists the readers] Final Index Scores Global Integrity takes full and final responsibility for the scores contained in the Integrity Scorecard for each country and the Global Integrity Index. These scores are generated following an elaborate and collaborative review process that includes balancing information from several (sometimes conflicting) sources while being guided by the master codebook. 10

11 While Global Integrity makes every attempt to produce credible information we welcome all feedback on the veracity and accuracy of our data. Please Global Integrity with specific comments on indicator scores that you may not agree with, particularly with regard to factual accuracy. In generating the final Index, initial data (both scores and narrative) gathered by the lead social scientist underwent several interactive rounds of substantive review over and above stylistic revisions and copyediting. These four phases of the review process served to strengthen the integrity of the Index as a research tool. They may be summed up as follows: 1) Interpretation. The scoring committee reviewed commentary provided by the lead social scientist to ensure a uniform application of each indicator. Because highly specific questions are asked in diverse contexts, accurate application of a consistent approach is crucial to obtaining reliable data. This process involved dialogue with the lead social scientist and was repeated during later review stages. 2) Peer review. After editing and corrections based on interpretation, each dataset was posted online for comment from a panel of carefully selected researchers and journalists. These comments indicated which scores panelists disagreed with and included explanations for their dissent. The entire process was double-blind: The lead social scientists did not know who was reviewing their work, and the reviewers did not know on whose research they were commenting. Reader comments were used to interpret and in some cases adjust scores that reviewers identified as erroneous, biased, or out-of-date. In order to adjust a score, Global Integrity followed a number of rules, notably requiring similar comments from multiple reviewers and the provision of specific, solid references to settle disagreements over questions of fact. 3) Indicator evaluation. Based on feedback from the field teams and the double-blind peer reviews, several indicators were cut from the final report. Only indicators judged to be distinct, universally valid, and reliable were included in the final set of Integrity Indicators. 4) Intercoder reliability. Personality, language, and culture can all affect the relationship between the qualitative interpretation of a particular indicator and the score assigned to it. To minimize this effect, Global Integrity examined each indicator for consistency among scorers. Reviewing the codebook scoring criteria and reader comments, the scoring committee examined the full set of scores; in some cases, score adjustments were ultimately made to improve intercoder consistency. 11