EVALUATION NOTE. Evaluating Trickle Up s Graduation Programs in India. Findings from a quasi-experimental evaluation in West Bengal and Jharkhand.

Transcription

1 EVALUATION NOTE Evaluating Trickle Up s Graduation Programs in India Findings from a quasi-experimental evaluation in West Bengal and Jharkhand.

2 INTRODUCTION In 2012, the Ford Foundation supported Trickle Up to conduct a 3-year Graduation project with 900 households living in extreme poverty in the Indian states of West Bengal and Jharkhand. This evaluation brief summarizes results and learning from this project, including from a quasi-experimental evaluation in West Bengal. View the full report is available at Our analysis suggests that participants were able to achieve a wide range of economic and social outcomes that are indicative of being on a sustainable pathway out of poverty. In most areas, the gains made by participants in West Bengal were significantly and considerably more than those of a comparison group: they have made significant improvements in income, assets, livelihood diversification, financial inclusion, food security, and personal and social empowerment. CONTEXT More than one fifth of the world s population lives in extreme poverty, surviving on less than US $1.901 a day. This population is often characterized by insecure livelihoods and few productive assets, and they often live in geographically isolated areas with substandard or no infrastructure and weak markets. There is growing international consensus that this number of households can and must be reduced to zero by Building sustainable livelihood options for these last mile families those most difficult to reach and at greatest 1. At purchasing power parity 2. World Bank, The World Bank Annual Report 2013 (Washington, D.C., 2013). 1 Evaluating Trickle Up s Graduation Programs in India

3 risk of being left behind is a critical component in the effort to eradicate extreme poverty. The Graduation Approach to livelihood development, originally developed by BRAC in Bangladesh and since piloted in a variety of countries, has proven to be successful in providing a pathway out of extreme poverty for the most vulnerable. Randomized control trials (RCTs) of six Graduation pilots provide compelling evidence that this approach can help extremely poor households make significant and lasting improvements across a range of economic and social indicators. 3 Trickle Up s experiences supporting the poorest of the poor through an approach that is rooted in a strong evidence base has led to partnerships with two state governments in India to integrate the Graduation Approach into their large scale poverty alleviation programs. As we undertake these scale opportunities, we will continue to draw on lessons learned from other livelihood practitioners and will share our own learnings in an effort to contribute to the growing evidence base on Graduation. To that end, this evaluation brief summarizes findings from a Graduation project with 900 women and their families in West Bengal and Jharkhand, and presents areas for further investigation. 3. A multifaceted program causes lasting progress for the very poor: Evidence from six countries. Abhijit Banerjee, Esther Duflo, Nathanael Goldberg, Dean Karlan, Robert Osei, William Parienté, Jeremy Shapiro, Bram Thuysbaert, and Christopher Udry, Science 15 May 2015: 348 (6236), [DOI: /science ] Evaluating Trickle Up s Graduation Programs in India 2

4 PROJECT SUMMARY With support from the Ford Foundation, Trickle Up India partnered with local organizations Jamgoria Sevabrata in West Bengal and Network for Enterprise Enhancement and Development Support in Jharkhand to implement a Graduation project with 900 women and their families between 2012 and Our goal was to enable women participants and their households to develop the skills, resources and connections to grow and sustain livelihoods that move and keep them out of extreme poverty. The Graduation project consisted of a carefully sequenced combination of activities, each designed to address specific constraints facing households living in extreme poverty. The Graduation Approach 1. Poverty targeting and participant selection: Community participatory wealth ranking exercise to identify the poorest households and household-level verification. 2. Self-help group (SHG) formation: Integration of participants into savings and lending groups to promote financial inclusion, provide financial education, and help participants build social capital. 3. Livelihood planning and training: Technical training and support to plan and manage the investment of seed capital. 4. Seed capital grant transfer: A transfer of Rs. 11,280 ($205) to jumpstart planned livelihood activities. 5. Regular coaching and monitoring: Weekly to bi-weekly visits by field agents to reinforce training, problem solve, and provide encouragement. 3 Evaluating Trickle Up s Graduation Programs in India

5 6. Consumption support: Consumption stipend for two months during the first lean season to protect against the liquidation of productive assets and reduce the need to migrate. Pregnancy allowances for expecting mothers are also provided. 7. Health promotion: Monthly visits by health workers to provide preventative health information and encouragement to access public health services. EVALUATION OVERVIEW The evaluation aimed to assess the outcomes and estimated impact of Trickle Up s Graduation program in two rural sites in India, and to situate the findings within those from the RCTs on the Graduation pilots presented by Banerjee et al. This report draws on a variety of data sources, including quasiexperimental data to assess outcomes for participants in West Bengal relative to a comparison group, and pre-post participant outcomes in Jharkhand. We also draw on program monitoring data and focus group discussions. In all cases, our analysis has been informed by consultations with local partners and participants themselves. Program participants were selected through a community participatory wealth ranking (PWR) exercise, followed by household-level verification. A comparison group was selected (in West Bengal only) from within the same communities, by drawing on data from the participant selection process. The comparison group consisted of households that were identified Evaluating Trickle Up s Graduation Programs in India 4

6 as among the poorest in their communities through PWR exercises, but were disqualified from program participation after household-level verification, largely due to having too much land or other assets. In order to estimate the impact of Trickle Up s program, while taking into account baseline differences between the participants in West Bengal and comparison group, we utilized regression discontinuity to exploit the wealth ranking of the two groups and difference-in-difference to assess changes in outcomes between the participant and comparison groups over time. The full report contains more details on the research design and selection of the comparison group. Lucy Hansda at her stall in Jharkhand, India 5 Evaluating Trickle Up s Graduation Programs in India

7 PROGRAM FINDINGS 1 Livelihood Development & Economic Impacts 1.1 Income Participant households increased their total annual income by an average of Rs. 29,000 ($527), an estimated Rs. 11,300 ($205), four of which can be attributed to the program. Participant households in West Bengal started with a slightly lower annual income, yet by the end of the project, their income was nearly 25% higher than the comparison group. This difference is highly statistically significant. Specifically, participants increased their total household annual income by approximately Rs. 29,000 ($527), mainly due to agricultural, small business, and livestock income, which were the types of Average Annual Income by Source 50,000 45,000 40,000 35,000 30,000 25,000 20,000 15,000 Other Income Small Business Income Livestock Income Agricultural Income Wage: NREGA Wage: Non-agriculture Wage: Agriculture Wage: Migration 10,000 5,000 0 WB Comparison (Baseline) WB Project (Baseline) WB Comparison (Endline) WB Project (Endline) 4. Throughout the report, we used the average exchange rate between the Rupee and US Dollar between the years 2012 and 2015 Rs. 55: USD 1 Evaluating Trickle Up s Graduation Programs in India 6

8 activities promoted by the project. This compares to an increase of Rs. 17,700 ($322) for the comparison group, suggesting that Rs. 11,300 ($205) of increased income for program participants can be attributed to the program. However, qualitative assessment suggests positive spillover effects on the community, as many successful agricultural practices adopted by participants were replicated by members of the comparison group. This implies that the true estimated project impact may be even greater. We did not measure income for participants in Jharkhand. These findings are in accord with the results presented by Banerjee et al. In those studies, the overall largest (standardized) effect across all sites was on an index of income and revenues. 1.2 Assets Participant households increased the total average value of household, land, and livestock assets by Rs. 34,000 ($618), an estimated Rs. 14,000 ($254) of which can be attributed to the program. 45,000 40,000 35,000 30,000 25,000 20,000 15,000 10,000 5,000 Total Average Value of Assets (Rs.) 0 Baseline Endline WB Comparison WB Project JH Project 7 Evaluating Trickle Up s Graduation Programs in India

9 The total average value in household, land, and livestock assets (excluding saving) increased substantially for both participant groups. Participants in West Bengal increased their total average asset value by Rs. 34,000 ($618) by the end of the project, compared to an increase of Rs. 20,000 ($364) for the comparison group. The difference is statistically significant, and implies that the estimated program impact is an average increase of Rs. 14,000 ($254) in asset value. These findings align with those of Banerjee et al., who found large impacts on total assets and productive assets, and smaller but positive impacts on household assets. 1.3 Diversification of Livelihood Activities & Wage Labor Participant households diversified their livelihood activities by an average of 1.5 extra distinct occupations. An estimated average increase of 0.8 occupations can be attributed to the program. Participants also reduced reliance on daily wage labor significantly more than the comparison group. Livelihood diversification is important for increasing household resilience to external shocks and for building alternatives to daily wage labor, which is often sporadic and unreliable. On average, each participant household in West Bengal increased from 2.2 occupations to 3.7, for an average of 1.5 extra distinct occupations at the end of the program. In contrast, the comparison group increased by 0.7 occupations. This difference between the two groups was highly statistically significant, and suggests that the program was responsible for an average increase of 0.8 Evaluating Trickle Up s Graduation Programs in India 8

10 extra occupations. Participants in West Bengal also reduced their reliance on daily wage labor significantly more than the comparison group. Such labor comprised the reported primary occupation for 66% of participants at baseline, and fell to 3% by the end of the program. Ninety-five percent of participants in West Bengal and Jharkhand engaged in two or more livelihood activities at the end of the project. Livelihood diversification was not discussed by Banerjee et al.; however they did note an increase in time spent on livestock and agricultural activities (but no impact on wage labor), which is broadly consistent with Trickle Up s findings. 1.4 Migration Participant households significantly reduced migration by any family member. An estimated decrease of 55 percentage points can be attributed to the program. Percent of Households with a Member who Migrated in the Past 12 Months 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Baseline Endline WB Comparison WB Project JH Project Inconsistent incomes, especially during the lean season, can force families to engage in distressed migration, which is often associated with hazardous living and working conditions, risks of 9 Evaluating Trickle Up s Graduation Programs in India

11 sexual exploitation, and long absences from school for children. Participant households in West Bengal were 50% more likely to have a member who migrated at baseline than the comparison households. However, by the end of the project, migration by any family member for households in West Bengal reduced from 90% of households to 22%, for a total decrease of 68 percentage points. In contrast, migration for the comparison group households reduced by 13 percentage points. This difference is statistically significant, and suggests an estimated project impact of 55 percentage points. The dramatic reduction in migration was also seen in households in Jharkhand. Migration was not discussed by Banerjee et al. 2 Financial Inclusion 2.1 Households with Savings The number of participant households with savings increased significantly. An estimated increase of 45 percentage points can be attributed to the program. After completing the project, 99% of participant households in both West Bengal and Jharkhand had savings (up from 10% in West Bengal and 0% in Jharkhand). This was an increase of 89 percentage points for households in West Bengal, which was significantly different from the comparison group, who saw an increase of 44 percentage points. This implies an estimated project impact of 45 percentage points. Further, 99% of participant households had at least Rs ($36) in savings in SHGs, which confirmed that the project was very successful at supporting participants to save, even in small quantities. Evaluating Trickle Up s Graduation Programs in India 10

12 Importantly, qualitative data suggest that participants are highly committed to continuing their SHG participation; however, future follow-up is required. Participants in West Bengal significantly increased their savings in SHGs, from an average of Rs. 320 ($6) at baseline to Rs ($73). We are unable to directly compare savings quantities between the two groups in West Bengal, given differences in savings data sources. 2.2 Loans from Moneylenders Participant households significantly reduced their reliance on moneylenders for loans. An estimated decrease of 24 percentage points can be attributed to the program. There was a significant reduction in the number of program participant households in West Bengal and Jharkhand with debt from a moneylender. We found a decrease from 25% of households in West Bengal at baseline to 2%, for a total reduction by 23 percentage points. In contrast, the number of comparison group households with loans from a moneylender increased by 1 percentage point, resulting in an estimated project impact of 24 percentage points. We found that participant households shifted away from moneylenders to lower interest loans from SHGs. The purposes of debt tended to shift from consumption and health care to more productive investment in livelihoods. These results generally align with the findings from Banerjee et al.; however, they found that the pilot projects increased loans from formal financial institutions, while no effect was found on the amount borrowed from informal sources. This variation can be explained by the difference in approach to financial inclusion. The Graduation pilots were largely implemented by micro-finance institutions and promoted linkages to their 11 Evaluating Trickle Up s Graduation Programs in India

13 formal financial services, whereas Trickle Up promotes SHGs (considered to be informal) as a primary entry point for financial inclusion and means by which participants eventually link to formal institutions. 3 Food Security Having Enough to Eat Participant households significantly reduced the frequency of food insecurity. An estimated decrease of 22 percentage points can be attributed to the program. The number of participant households in West Bengal and Jharkhand who reported sometimes or often not having enough to eat in their household over the past year reduced from 45% and 57% respectively at baseline to only 1% of households at endline. The decrease by 44 percentage points for participant Sometimes or Often Did Not have Enough To Eat 70% 60% 50% 40% 29% 45% 57% 30% 20% 10% 0% Baseline 7% 1% Endline 1% WB Comparison WB Project JH Project Evaluating Trickle Up s Graduation Programs in India 12

14 households in West Bengal is significantly different from the decrease of 22 percentage points experienced by the comparison group, suggesting an estimated project impact of 22 percentage points. We also found a considerable reduction in the reported number of months of food scarcity in West Bengal; however this change was not significantly different from the comparison group and requires further assessment. These improvements in food security align with the findings from the six pilots reported by Banerjee et al.: significant increases were reported across four of the six sites, particularly in India and Ethiopia. 4 Social Empowerment 4.1 Decision-making Participants significantly increased their involvement in household decisionmaking. An estimated increase of.24 points (out of 1) on an empowerment index can be attributed to the program. We assessed household decision-making across five domains children s education, healthcare, family planning, housing, and finances plus an index combining these all into a single empowerment metric. Participants in West Bengal were less empowered than the comparison group at baseline, yet by the end of the project, the women significantly increased in all decision-making domains, relative to the comparison group. We found an index increase of.35 for participants, versus.11 for the 13 Evaluating Trickle Up s Graduation Programs in India

15 comparison group, suggesting an estimated project impact of.24. This pattern of increased decision-making was consistent among Jharkhand participants as well, and the biggest impact for all participants was on financial decision-making Decision-making Index 0 Baseline Endline WB Comparison WB Project JH Project These findings differ substantially from the article by Banerjee et al., which found no overall effect on a very similar women s empowerment index or any of its components. The researchers found that women s empowerment was affected slightly at the time of the first follow up (which corresponds to the endline data presented here), but had diminished to become statistically insignificant one year later. While the focus on women as primary participants and the addition of a gender justice component to Trickle Up s program may have increased the impact on women s role in decision-making, it will be important to assess this change over a longer period. 4.2 Collective Actions Participants significantly increased their engagement in collective action. Evaluating Trickle Up s Graduation Programs in India 14

16 An estimated increase of 49 percentage points can be attributed to the program. No significant increase was found in Panchayat participation. We found a dramatic increase in the number of participant households that engaged in collective action, from 1% to 92% of participants in West Bengal and 5% to 93% of participants in Jharkhand. While the comparison group also increased by 42 percentage points, the difference in change between the two groups was highly significant. This suggests an estimated project Percent of Households that Engaged in Collective Action in the Previous Year 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Baseline Endline WB Comparison WB Project JH Project impact of 49 percentage point. Participants are encouraged to engage in collective actions through their SHGs, and the most common form of action was to advocate for community work or infrastructure improvements from the government, such as roads or irrigation systems. Participants also increased their participation in Panchayat meetings (India s lowest level of government); however, this increase was not significantly different from the comparison group, and contextual factors, such as increased outreach by new political leaders, are believed to have played an important role in this increase for both groups. 16 Evaluating Trickle Up s Graduation Programs in India

17 Banerjee et al. found that project participation increased political involvement in four of six sites. This is consistent with our findings on collective action, though our data on Panchayat participation suggest no significant impact. 5 Access to Social Services Point of Treatment for Healthcare Participant households significantly reduced their reliance on informal rural health practitioners as a primary point of treatment, and shifted in favor of using formal health services. An estimated decrease of 16 percentage points can be attributed to the program. In West Bengal, we found a significant reduction in reported usage of rural medical practitioners as a primary point of treatment, relative to the comparison group. While the participant group decreased from 10% of households to 6%, for a total reduction of 4 percentage points, the number of comparison group households actually increased by 12 percentage points, for an estimated project impact of 16 percentage points. This shift away from rural practitioners to formal health services is promoted through messaging and outreach. The change was largest for the participants in Jharkhand, which our partner, NEEDS, believes is primarily due to their active collaboration with the National Health Mission to Evaluating Trickle Up s Graduation Programs in India 17

18 deliver health messaging, education, and linkages. At the end of the project, 93% of participants in West Bengal reported a public health center as their primary point of treatment for their families. Health access specifically was not examined by Banerjee et al.; however they did measure a number of health-related outcomes and found little project effect across all sites. LESSONS LEARNED Evaluation results suggest that participants have made substantial and significant improvements in income, assets, financial inclusion, food security, and personal and social empowerment, changes which are largely consistent with the RCT findings from the Graduation pilots. However, further assessment is required, particularly in the areas of financial inclusion, food security, and political participation, in order to deepen our understanding of the mechanisms by which these gains were achieved by all groups, including potential spillover effects and contextual factors. We also aim to assess changes over a longer period, in order to explore the sustainability of outcomes. This project has not only provided Trickle Up with new experiences to draw upon as we further refine our implementation of the Graduation Approach, but it has also been instrumental in positioning Trickle Up to enter into partnerships with the state branches of the Indian government s National 18 Evaluating Trickle Up s Graduation Programs in India

19 Rural Livelihood Mission (NRLM) in Jharkhand and Odisha. The integration of Graduation into large scale social protection and poverty alleviation programs is necessary in order to make a significant impact on extreme poverty, and with this new phase comes further questions and challenges. Together with our NRLM partners, we will assess the added value of the Graduation Approach when integrated into India s flagship livelihood development programs. As we prepare for these scale opportunities, we have identified a number of key lessons from our evaluation and experiences to date that will be particularly important to act on and explore further: Capitalize on the role of self-help groups as a foundation for development. SHGs appear to play an important role in building the social and financial capabilities of participants in Trickle Up s Graduation program, and in sustaining them after graduation. In India, the SHG also acts as a critical foundation for a number of large government programs to deliver livelihood development services and other complementary benefits. The sustainability of SHGs is not guaranteed after the program ends, however. Working with the government to cluster SHGs into federations holds the potential to both strengthen their institutional capacity in the long term, while also bolstering their ability to access other resources and programs individually that they may lack the leverage to achieve. We will continue to assess the role of these groups as a mechanisms for sustainable livelihood development. Promote more strategic linkages to government programs during the livelihood planning process. We found noteworthy innovations when field agents went beyond supporting participants to access employment and other benefits, to also strategically incorporating government programs for community infrastructure into the livelihood planning phase (such as for irrigation). This allows staff to increase the menu and profitability of livelihood activities for participants, which may otherwise be deemed unfeasible without certain types of infrastructure. The challenge is to ensure that staff have Evaluating Trickle Up s Graduation Programs in India 19

20 the knowledge and skills to utilize these services, while also educating and empowering participants themselves to advocate for access to such programs. Strengthen the enabling environment for the poorest through broader community engagement. Through this project, we identified a need for interventions with an additional target population: households that are barely above the threshold of extreme poverty, such as the comparison group, and who are at risk of falling into extreme poverty. We are currently exploring the impact of a lighter Graduation Approach (specifically, the elimination of the grant transfer) with these households. Working with both groups allows us to reach a significantly larger number of participants, as the ratio of non-grant to grant participants is currently 2:1. We hypothesize that this helps to create an enabling environment for extremely poor households in three ways. First, it establishes a critical mass necessary for the gradual uptake of behavioral changes, such as improved health and gender-empowerment practices. Second, it creates a larger platform for community-level collective action, such as joint advocacy for a new road. Finally, it creates a population that draws the interest of local politicians and provides a stronger base for policy reach to the more vulnerable. Enhance the cost-effectiveness and scalability of the program. Coaching appears to have been critical for reinforcing training and providing motivation for participants; however, we also know that coaching is the most challenging component to scale and constitutes a significant program cost. We aim to test different configurations of delivering coaching, such as through community resource people, who are women from the community that can serve as an advocate and resource for program participants. We are also exploring the use of mobile technologies to support monitoring and management, including the ways in which these technologies can potentially reinforce training and trouble shooting. drawing on data from the participant selection process. The comparison group consisted of households that were identified 20 Evaluating Trickle Up s Graduation Programs in India

21 ORGANIZATIONAL BACKGROUND Trickle Up Trickle Up was founded 1979 to bring livelihood opportunities to people living in extreme poverty and has since reached a million people worldwide. Trickle Up works in India, Central America and West Africa to support the most vulnerable populations to create a sustainable pathway out of extreme poverty through livelihood development programming rooted in the Graduation Approach, and provides technical assistance to other organizations to integrate the Graduation Approach into their service provision and policies. The Network for Enterprise Enhancement and Development Support (NEEDS) NEEDS is a community-based NGO that has been working in Jharkhand, India since 1998, to promote food security, sustainable livelihoods, reproductive health and child protection with people living in poverty, particularly in tribal communities. Jamgoria Sevabrata (JS) JS is a rural development organization that was established in 1986 in India to promote livelihood opportunities for poor and marginalized communities through land and water management, credit linkage with banks, and the promotion of income-generating activities. The Authors Ameneé Siahpush, Jo Sanson, & Matthew Bombyk For the full evaluation report and more information on our programs, please visit Evaluating Trickle Up s Graduation Programs in India 21

22 India Regional Office 4/2 Orient Row, Park Circus Kolkata INDIA New York Headquarters 104 W 27th Street, 12th Floor New York, NY US / info@trickleup.org