Experiments and Beyond Local Economy-wide Impact Evaluation J. Edward Taylor University of California, Davis Agricultural & Resource Economics Berlin, November 6, 2015
Consider a Social Cash Transfer (SCT) Program Cash payments to ultra-poor households Inspired by Mexico s PROGRESA program Eligibility: Asset poor, labor poor, other criteria (e.g., orphans and vulnerable children) Conditionality: Kids enrollment in schools, clinics (PROGRESA) Most Africa SCTs do not have hard conditionalities
Cash Transfer Programs in Sub Saharan Africa (19 in 13 Countries) Malawi SCT* Mchinji pilot, 2008-2009 Expansion, 2013-2014 Kenya* CT OVC, Pilot 2007-2011 CT OVC, Expansion, 2012-2014 HSNP, Pilot 2010-2012 Mozambique PSA Expansion, 2008-2009 Zambia* Monze pilot, 2007-2010 Child Grant, 2010-2013 South Africa CSG Retrospective, 2010 2012 Ethiopia* PNSP, 2006-2010 Tigray SPP, 2012-2014 Ghana LEAP* Pilot, 2010-2012 Lesotho, CGP* Pilot, 2011-2013 Uganda, SAGE Pilot, 2012-2014 Zimbabwe, SCT* Pilot, 2013-2015 Tanzania, TASAF Pilot, 2009-2012 Expansion, 2012-2014 Niger Begun in 2012 * Local-economy (LEWIE) included in evaluation; see http://www.fao.org/economic/ptop/home/en/
The Experiment (1) Identify the eligible households Proxy means tests (PMT), community-based assessment (CBA) (2) Conduct a baseline survey with variables of interest to evaluation (3) Randomly select treatment and control villages (4) Roll out treatment to treatment villages (withhold from controls) (5) Conduct follow-on surveys (6) Compare outcomes of interest in treatment and control villages
An Illustrative Example (Outcome = Income) Treated Villages Monthly Difference Income Eligible Households Before 100 40 After 140 Control Villages Monthly Difference Income 100 110 10 Difference in Difference 30 We might be able to say the SCT program caused the $30 change in income That is, that the average effect of the treatment on the treated (ATT) is 30 Two critical conditions to establish causality in this project: Treatment and control villages are random (randomized control trial, RCT) There are no spillovers between treatment and control villages (control-group contamination) Not a good idea to draw treatment and control groups from same village External validity: Nonstarter without knowing why the outcomes happened
SCT $ Feedback on the Treated? $ Eligible Household in Treated Village $ $ Spillovers to Ineligibles $ $ $ Rest of Zimbabwe Control Village?
We Could Use an Experiment to Explore Spillovers Treated Villages Monthly Difference Income Eligible Households Before 100 40 After 140 Difference in Difference Control Villages Monthly Difference Income 100 110 30 10 Ineligible Households Before 200 After 220 Angelucci and DiGiorgi, Mexico s PROGRESA (2009) In practice, almost no project does this 20 Difference in Difference 10 200 210 10 Could mean a big total spillover (if there are many ineligible households)
Where Do SCT Income Spillovers Come From? Cash raises purchasing power of beneficiary households thus demand in the local economy supply must rise to meet this demand (otherwise, inflation) Economic impacts depend critically on the supply response If local supplies expand to meet demand, SCTs can have a multiplier effect on local income Otherwise the program could be inflationary Different kinds of interventions can produce different kinds of spillovers (including non-economic ones)
What Is Local Economy-wide Impact Evaluation (LEWIE)? A Simulation Approach, Complement to Econometric models RCTs Precedents: Social accounting matrix (SAM) analysis Computable general equilibrium models Model of How Treated Economies Work Micro-actors (beneficiaries, non-beneficiaries) interact within a LEWIE model Construct with micro-survey data Econometrics estimates give confidence Can simulate different policy interventions
Steps to Doing (or Commissioning) a LEWIE Scoping mission: structure of local economy Production activities How do local markets work? Markets convey impacts through local economies Modify baseline surveys for LEWIE Include both eligible and ineligible households Data to econometrically estimate household expenditure, production functions The where question May have to supplement household surveys with business surveys Construct models of eligible and ineligible households Draws from rich experience constructing agricultural household models Integrate these into LEWIE model of local economy (LEWIE) (GE modeling tools) Use the LEWIE model to simulate impacts of intervention, test sensitivity to model assumptions, obtain confidence bounds on simulation results Optional: Explore complementary interventions Typically have LEWIE results 4-6 months after baseline survey is complete
Why Is LEWIE Important? Lets us identify total local economic impacts, including spillovers These are part of the cost-benefit analysis of the project It s an option when experiments are not feasible There is no treatment and control group Politics, ethics, implementation challenges, cost It provides explanations for why impacts happen For policy, it s not enough to know what the impacts are We need to learn why and how to influence them We want to spot and deal with possible negative impacts Timing: LEWIE can be done before the project is implemented and inform project design
LEWIE Estimates of SCT Income Multipliers in Seven African Countries* Cash transfers to the poor create local income multipliers A dollar transferred to a poor household raises local income by significantly more than a dollar * FAO, Protection to Production (PtoP) Project; http://www.fao.org/economic/ptop/home/en/
Impact LEAP has had a positive impact on local economic growth. Beneficiaries spend about 80 percent of their income on the local economy. Every GH1 transferred to a beneficiary has the potential of increasing the local economy by GH2.50. - President John Dramani Mahama, opening the Pan- African Conference on Inequalities, April 2014
Where Do We Find the Largest Spillovers? High there is high reliance on local supplies of goods and services Where the local supply response is high Or complementary interventions make supply response more elastic Real impacts are lower in more labor, capital, land, liquidity constrained contexts Increase in demand translates into increase in supply for local goods and services Prices transmit impacts More price action in local economies isolated from outside economies, more nontradables More integrated with outside markets, more widely we have to cast our net to find impacts
Most LEAP SCT Spillovers Go to Non-beneficiary Households Nominal Real 2.5 2.5 2 2 1.5 1.5 1 Spillover Transfer 1 0.5 0.5 0 0 Total Beneficiary Non-beneficiary Total Beneficiary households Non-beneficiary households households households Many non-beneficiaries are poor, just not poor enough to quality Fail to meet other eligibility criteria (LEAP: OVC, elderly, disabled) FAO, Protection to Production (PtoP) Project; http://www.fao.org/economic/ptop/home/en/
A Diversity of LEWIEs Country Impact of References Tigray Social Cash Transfer Pilot Program* Kagin, et al. (2014); Davis, et al. (2016) Ethiopia Evaluation of economy-wide impacts of Productive Safety-Nets Program (PSNP) In Progress, 3ie-IFPRI Cost-benefit Analysis of SCTs In Progress, UNICEF Galapagos Impacts of Ecotourism Taylor, et al. (2009), (2003), IADB Livelihood Empowerment Against Poverty Ghana Program (LEAP)* Davis, et al. (2016) Rural Income Transfers Compared Filipski and Taylor (2012), UNICEF Lesotho Child Grants Program (CGP)* Filipski, et al. (2015); Davis, et al. (2016) Experimental Validation of LEWIE Results In Progress, FAO-UNICEF Cash Transfer Program for Orphans and Kenya Vulnerable Children (CT-OVC)* Thome, et al. (2013); Taylor, et al. (2013) Hunger Safety Nets Programme Phase 2 In Progress (UKAID) Social Cash Transfer Program (SCT)* Thome, et al. (2014); Davis, et al. (2016) Malawi Fertilizer Input Subsidy Program (FISP) Thome, Taylor and Filipski (2014), SOAS Rural Income Transfers Compared Filipski and Taylor (2012), UNICEF Migrant Remittances Taylor and Filipski (2014) Hewlett Mexico Maize Price Shocks Dyer and Taylor (2011) Corruption in PROCAMPO Delivery Taylor and Filipski (2014) Morocco Saffron Price Shocks, Gender Taylor and Filipski (2014), CIMMYT Philippines Regulation of Small-scale Fisheries In Progress, Packard Rwanda Cash Programs in Refugee Camps In Progress, WFP Tanzania Technology Change in Cotton Production Kagin, et al. (2015), Gatsby Zambia Child Grant Program (CGP)* Thome, et al. (2014) Zimbabwe Harmonized Social Cash Transfer Program (HSCT)* Taylor, et al. (2014) * FAO Protection to Production (PtoP) Program, in collaboration with UNICEF
Beyond Experiments to Answer Big Questions (Mostly not RCTizable ) When poor women get cash, what happens to the local economy? Do non-beneficiaries (most of whom are likely to be poor) benefit? (Africa) Should the WFP shift from food aid to cash in refugee camps? Good for the refugees? Good for the host country? (Rwanda, Uganda) How can we make small-scale fisheries sustainable without adversely affecting fishing communities in the short-run? (Philippines) What happens when food price shocks hit poor rural economies? (C. Am) Are migrant remittances good for migrant-sending communities? (Mexico) If we make poor farmers more productive, are there benefits for the poor villages in which they live? (Tanzania, Ethiopia) Can eco-tourism development alleviate poverty in poor regions even though tourists rarely buy things from poor people? (Galapagos, Lat America) Which are better, cash transfers or fertilizer subsidies? (Malawi)
Each Evaluation Method Has Its Challenges and Drawbacks Experiments Feasibility, ethics, politics, timing Avoiding control group contamination (control villages beyond influence of the treatment Keeping control over the experiment and implementation bias Harder with big real-world projects, vs. academic ones Will people s answers determine whether they are eligible? Generalizing: Does the RCT give insight on what scaled-up programs would do? Tradeoff between internal and external validity Understanding why impacts happen RCTs test whether something works, not why Findings not available until some time after the project is implemented Econometrics & Quasi-experimental Methods Finding convincing instruments to control for selection bias; the identification problem
LEWIE Getting the model right Understand how local economy works Structural modeling draws from economic theory Rich literature on agricultural household models, imperfect markets Need good baseline data to estimate model parameters Overlap with experiment data needs Validation Confidence intervals on parameter estimates Tests for structural form Monte-Carlo method to put confidence intervals around simulation results Big advantage in LEWIE: the budget constraint People spend their income Most spend it close to home Potential for large local-economy spillovers and multipliers Rich history in economics of modeling expenditures, production, GE linkages
In Vivo in Vitro in Silico Shift in scientific research from in vivo/vitro to in silico methods* Rational drug design, human brain project, climate models LEWIE is in silico (RCTs are in vivo; econ lab experiments are in vitro) Like other in silico methods, it can benefit from experiments to obtain better parameter estimates update parameters that change as a result of a treatment validate simulation findings Contextual variation in how local economies work, but commonalities The budget constraint, supply and demand, role of prices Every $1 of SCT goes somewhere; similarities in similar contexts, often predictable across contexts Clear procedures to estimate expenditure and production functions * Witness a sharp increase in usage of in silico contrasted with declining use of in vivo and in vitro on the Google books Ngram viewer. A goal of the EU Human Brain Project is establish in silico experimentation as a foundational methodology for understanding the brain.
Learn by Integrating LEWIE with Experiments (A New Gold Standard?) LEWIE offers structural explanations for experimental findings Findings much sooner than experiments (do not have to wait for follow-on surveys to simulate impacts) Experiments can be used ex post to validate, update LEWIE findings Example: Lesotho s CGP Experimental data showed real-life multipliers are bigger than LEWIE simulated Likely: CGP changed the structure of the economy in ways not reflected in the model Local economy less constrained than we assumed Plenty of room to expand livestock in Lesotho New markets, consumption behavior, behavioral spillovers
Parting Thoughts: From Evaluation to Policy We see LEWIE as an increasingly important part of the evaluator s tool kit Governments and donors want to know impacts beyond the treated Including spillovers to ineligible households We can t affort to miss impacts of development interventions Part of the cost-benefit analysis Impacts on the treated include local-economy feedbacks The economic impacts of many projects are likely to significantly exceed the direct impacts (Often) good news for finance ministers and donors Structural: Focus on understanding why impacts happen; learn from evaluations Want to inform policy Why impacts happen, and how to influence them E.g., productive interventions to enhance impacts of SCTs LEWIE is a laboratory in which we can explore this Biggest impact of LEWIE: Changes the way people think about how their programs create impacts