Evidence from Randomized Evaluations of Governance Programs Cristobal Marshall Policy Manager, J-PAL December 15, 2011
Today s Agenda A new evidence based agenda on Governance. A framework for analyzing the importance of governance in development. Evaluating mechanisms to improve governance: Information Disclosures Monitoring Incentives 2
Using evidence to improve governance and thereby the development agenda Strengthening governance is both a means to improve development outcomes as well as an end in itself. Despite its importance most questions about how to improve governance remains unanswered. Recent advances have helped creating new evidence: New methods to quantify corruption. Improved methods to collect data in various contexts. The use of randomized evaluations to provide direct causal evidence on the drivers of good governance. The combinations of quantitative and qualitative work. 3
Good Governance can be framed as the relation between principals and agents Principal Agent Actions Citizens Politicians Citizens elect politicians to enact policies in line with their interests Governments, NGOs Civil servants, service providers Work with service providers to implement policies 4
But these links are often weak Politicians do not implement policies aligned with citizens preferences: Corruption Elite capture Service providers do not implement properly policies and programs. Lack of effort Absenteeism Bribe-taking 5
Strengthening these links via Information, Monitoring and Incentives Better Information can reduce adverse selection. Better Monitoring can increase accountability. Better Incentives can increase effort. 6
Mechanism 1: What is the Impact of Information? Evidence from Brazil and India 7
Exposing Corrupt Politicians in Brazil (Ferraz and Finan, 2008) Brazil implemented a federal anticorruption program that randomly selected municipalities to be audited. o Treatment group: 205 municipalities audited before the 2004 election. o Control group: 168 municipalities audited after the 2004 election. Publicly released the information of audits. Compare electoral outcomes of mayors who were audited before and after the election 8
Information Reduced Reelection Rates of Corrupt Mayors Publically released audits reduced likelihood of reelection for corrupt incumbent mayors by 7 percentage points. The effect was stronger in municipalities with local radio station. The audit policy in the presence of a local radio station also promotes noncorrupt incumbents. 9
Politicians Performance Information Campaign in India (Banerjee et al 2010) Used Right to Information Act to gather information on legislator spending, committee attendance, and participation in the legislature. Information was provided to slum dwellers in 200 randomly selected polling stations during the 2008 state legislature election in Delhi. Prominent local newspaper ran 2 report cards, belonging to neighboring jurisdictions, each day over 5 days. An NGOs disseminated pamphlets on legislator responsibilities and free copies of the newspaper 10
Positive Impact of Information on Voter Turnout 70% Effect of Report Cards on Voter Turnout 65% 0.61 60% 0.575 55% 50% Control Slums Treatment Slums 11
Negative Impact of Information on Vote Buying 0.7 0.6 0.625 Incidence of Cash Bribes 0.5 0.4 0.431 0.3 0.2 0.1 0 Control Slums Treatment Slums 12
Differential Impact of Information on Voting Behavior 0.08 0.07 Impact on incumbent vote share 0.069 0.06 0.05 0.04 0.03 0.02 0.01 0-0.01-0.0058 Median Legislator Best Legislator 13
Mechanism 2: Choosing the Right Monitoring: Evidence from Indonesia 14
Audits vs. Community Monitoring in Indonesia (Olken, 2007) Question: comparing top-down and bottom-up approaches to monitoring corruption Setting: 608 villages in Indonesia building road projects for a government program funded by World Bank. Decision making: Each village made a proposal for any combination of smallscale infrastructure. 15
Evaluation Design: Two alternatives approaches to monitor corruption 1. External monitoring: Increase ex ante probability of audits by government auditors from 4% to 100%. 2. Improving grass-roots monitoring: Invitations of hundreds of villagers to project accountability meetings. Anonymous comment forms distributed to villagers. 16
Outcomes: How to measure corruption? Outcomes: Missing expenditures in village roads Conducted engineering and price survey to estimate what village road actually cost to build. Compared to what contractors claimed road cost to build on official expense reports 17
Percent Missing Audits Reduced Corruption Effect of Audits on Percent Missing 30% 25% 20% Materials 15% Materials 10% 5% Wages Wages 0% Control Audits 18
Percent Missing Community monitoring had only a small effect on corruption Effect of Invitations on Percent Missing 30% 25% 20% 15% Materials Materials 10% 5% 0% Wages Control Wages Invitations 19
Understanding these results Why the impact of the audits was not larger? o Not a clear link between audits and punishment. o Audits could be combined with higher punishment. Why community monitoring was less effective? o Monitoring is a public good, but people are more likely to help when they benefit personally. o Free rider problem: reducing corruption on labor has clear benefits for some of the participants while benefits of reducing corruption on quantities was spread among the entire community. o Elite capture: 20
Mechanism 3: Linking Performances to Incentives and Monitoring 21
Monitoring and Incentives: Getting teachers to come to schools (Duflo et al 2010) Question: Can we use technology tools (ensuring mechanically that certain procedures are followed) and incentives to reduce leakages? Technology and Monitoring: In randomly selected schools, one student took a picture of the teacher and the class at the start and close of each school day. Incentives: Salaries for Randomly selected teachers were attached to attendance while teachers in comparison group received a flat salary. 22
The combination of monitoring and incentives decreased absenteeism and increased test scores Absenteeism decreased by 21 percentage point. Children s test scores increased by 0.17 standard deviations 23
But a Cautionary Tale An evaluation in India tested a combination of monitoring with incentives to nurses in rural health centers. Initially the program had a positive effect : 27.9 pp. increase in attendance. However, after six months there was no longer an impact: o Health administration allowed larger number of exemptions. o Nurses also tampered with the unit/time stamping machines. Lessons 1- Long run effects might be different: it could take corrupt officials time to learn how to manipulate a new system Lesson 2 - technologies are only as effective as the systems which process their information. Lessons 3 In some contexts, incentives should be automatically applied without giving blanket discretion to supervisors to relax conditions. 24
Many others completed or ongoing evaluations to test these mechanisms Information Monitoring Monitoring + Incentives Perceptions of female leaders in India. Can informational campaigns raise awareness and local participation in primary education in India. Information Dissemination Campaign and Voters' Behavior in the 2009 Municipal Elections in Mexico. The Impact of Information on Voter Knowledge and Engagement in Sierra Leone. Community-based monitoring of primary healthcare providers in Uganda. Increasing Effectiveness of Public Sector Nurses in India Through Monitoring Technologies. Low cost multifunctional device to reduce absenteeism in the health sector in India. Monitoring Patient Compliance with Tuberculosis Treatment Regimes in Pakistan. Understanding Incentives and Governance of Kenyan Coffee Cooperatives. Testing the Role of Wages, Incentives and Audit on Tax Inspectors' Behavior in Pakistan. The Role of Financial Incentives in Recruiting Public Sector Employees. Incentives for Nurses in the Public Health Care System in Udaipur, India. 25
Conclusions Strengthening governance is a means and an end. Information, monitoring and incentives are key mechanisms to improve governance. The key is in both the design and implementation of the programs and in understanding the specific contexts. Impact evaluations are a useful tool to understand what policies and programs works (or does not work) and why. 26