Evaluation Terms of Reference Refugee Crisis Response Program (BRCRP) Myanmar-Bangladesh Refugee Crisis Protection Project II November 2018-11-13 Project # B210368 World Vision Bangladesh Published October 2018, World Vision Bangladesh. 1
Table of Contents Project indicators... 5 2. Description of Programme summary... 5 Evaluation Target Audiences... 7 4. Evaluation Type... 7 5. Evaluation Purpose and Objectives... 7 8. Authority and Responsibility... 8 8.1 Team Members and Roles... 8 See appendix 1... 8 8.2 Partners (i.e., who will be involved in the evaluation?)... 8 9. Team Advisors... 8 10. Logistics... 8 11. Products... 9 13. Documents... 9 14. Lessons Learned... 9 15: Qualifications... 9 15. Appendices... 11 Published October 2018, World Vision Bangladesh. 2
i. Glossary ADH BRCRP CFW CSI DIP DPP DRR FGD IGA ISCG HH KII LEAP M&E NFI NO OSDM PDM SPSS TOR WFP WV WV BRCRP Aktion Deutschland Hilft Bangladesh Refugee Crisis Response Program Cash for Work Coping Strategy Index Detailed Implementation Plan Disaster Preparedness Plans Disaster Risk Reduction Focus Group Discussion Income Generating Activities Inter Sector Coordination Group Household Key Informant Interview Learning through Evaluation with Accountability and Planning Monitoring and Evaluation Non-Food Item National Office On-Site Distribution Monitoring Post Distribution Monitoring Statistical Package for Social Scientist Terms of Reference World Food Programme World Vision Bangladesh Refugee Crisis Response Program Published October 2018, World Vision Bangladesh. 3
ii. Introduction/Background In April 2018, World Vision Bangladesh partnered with Aktion Deutschland Hilft (ADH), Germany to implement the Myanmar-Bangladesh Refugee Crisis Project 11 over a period of 8 months from April 1 st, 2018 to November 31 st, 2018, with a project budget of US$1,185,864.30. The overarching goal of the project is to ensure the life-saving needs of the targeted 5, 780 vulnerable households. This was to ensure that the basic needs of both vulnerable Rohingya camp refugees and host communities are met and their suffering reduced. Bangladesh is experiencing one of the worst humanitarian crises in its history due to an unprecedented influx of refugees from Myanmar since August, 2017. Rohingya populations in Cox s Bazar and particularly in the sub districts of Ukhia and Teknaf have been present in villages since 1991. According to the Inter Sector Coordination Group (ISCG), as of January 2018 up to April 2018, 688,000 new arrivals joined with that of the refugees who arrived years ago raising the total number to 836,487. The UN Joint Response Plan, Mid Term Review (October 2018), reported an upward increase to 921,000 refugees from Myanmar living in Cox s Bazar, Bangladesh. Many of the refugees are living in 34 camps in Ukhia and Teknaf sub-districts under Cox s Bazar district creating a huge impact on the lives and livelihoods of the now minority host communities. The key priorities for the humanitarian sector in Bangladesh providing assistance in Rohingya Camps has been the need to ensure there is a sustainable food pipeline to meet the daily needs and to meet the dietary diversity for refugees and the host population. Malnutrition among children, pregnant and lactating mothers is quite high. It was upon this background that this project was implemented. 1. Evaluation Summary Programme/Project Programme Phase Evaluation Type Evaluation Purpose Primary Methodologies Myanmar-Bangladesh Refugee Crisis Protection Project Phase II Impact Assessment for the BRCRP project To determine how well the project s design addressed needs and priorities of the targeted communities the evaluator should ascertain changes attributed to the project focusing on successes, challenges, intended and unintended impacts resulting from the implementation of the project. Additionally, assessment will capture promising practices and key and lessons learnt and provide action orientated recommendations to inform similar future projects. Literature review, Key Informant Interviews, Focus Group Discussions, Household Survey Questionnaire, Case studies/impact stories/stories of significant change and any other data collection methods agreed between the consultant and World Vision Bangladesh assessment team Proposed Dates of Impact 7th to 25th January 2019 Assessment Submission of the inception report 7th January Field Work/Data Collection 14th to 24th January 2019 Date of Draft Impact Assessment Report 31st January 2019 Published October 2018, World Vision Bangladesh. 4
Submission Date of Final Impact Assessment Report Submission 14th February 2019 Project indicators Project Objective Summary Indicators Life-saving needs are met and suffering reduced for Rohingya refugees of Myanmar and host communities in Bangladesh Results (concrete expected Results) Results (concrete expected Results) Improved gender- and culturally-appropriate basic livelihood services for most vulnerable refugees and host community populations, particularly girls, boys, and women Increased disaster preparedness, resilience, and strengthened positive coping strategies for vulnerable refugees and host communities # beneficiaries received NFI kits # beneficiaries received nutrition and non-food vouchers # beneficiaries implement IGA # beneficiaries that have been engaged through Cash for Work # of beneficiaries covered by DRR schemes # HH reported higher Coping Strategy (CSI) Index score disaster DPP developed with local government # of sites improved (including landslide protection, drainage and waste management sites) in 3 camps and 2 unions for reducing disaster risks in project working areas 2. Description of Programme summary The project s strategy was integrated, which included the provision of Non-Food Items (NFI) kits, the provision of fresh food vouchers to meet the basic nutrition requirements, the provision of agricultural inputs for homestead gardening and training for the host communities, provision of inputs and trainings to support new/alternative livelihood skills and Income Generating Activities (IGA) opportunities to host community families, the provision of cash grant/support for livestock and poultry rearing for the host community. The project s package also included the implementation of community-level risk reduction schemes to the existing infrastructure or facilities (drainage, shelter, access road) mainly to mitigate against to future disasters. This was achieved through the Cash for Work component of the program. Setting up the Disaster Management Committee within the existing structures in both refugee and host communities settings) and strengthening the Disaster Preparedness Plans (DPP) and capacity of the community was also a priority. Project Goal: Life saving needs are met and suffering reduced for Rohingya refugees of Myanmar and host communities in Bangladesh Published October 2018, World Vision Bangladesh. 5
Outcome 1: 5780 HHs from Rohingya and host communities in Bangladesh to meet their basic needs Output 1: Improved gender and cultural appropriate basic livelihood services for most vulnerable refugees and host community particularly girls, boys and women To ensure that the targeted population s social services were met the project distributed NFI kits amongst to 1100 vulnerable HHs in refugee communities. The distributed package included solar lights, umbrella, utensils, cooking fuels, hygiene materials (soap etc), which were distributed as per the UN/ISCG cluster guidelines. As informed by the need assessment, which was carried out in Taknaf and Ukhiya, the organization distributed 3577 fresh food vouchers to newly arrivals that were identified as vulnerable mostly female headed and ultra poor HHs, ensured that food consumption basic needs were meet. Additionally, 55 USD support vouchers were distributed for three months to cater for urgent/emergency needs. Under livelihoods, the project supported 2600 HHs with agricultural inputs to carried various garden farming such as garden/ pit crop/ sack gardens and keyhole gardens by host communities prior of which orientation sessions were carried out to equip the benefitting HHs. Furthermore, 1000 vulnerable HHs amongst the host communities were identified and supported with nonfarm IGAs which included crops, livestock, small business management, saving s and group development. Capacity sessions were provided as part of capacity development. An alternative IGA intervention was also provided in the form of inputs and cash grants as well as technical support mainly to support household income. Output 2:- Increased disaster preparedness, resilience, and strengthened positive coping strategies for vulnerable refugees and host communities The project also focused on disaster preparedness through stakeholder coordination where a vulnerability and community resilience mapping was done. For risk reduction initiatives both communal and host community s sites, the project upgraded and constructed structures with necessary materials using skilled and unskilled labor force were supported through Cash for work for both host and refugee community. In this regard pathways leveling, drainage and waste management, communal center construction and coordination meetings were held with other agencies. 1000 vulnerable households in the refugee and 1109 HHs of host community did cash for work activities and which had ultimate contribution in impacting their positive coping strategies. It also done of 22 street lights reaching 800 households and impacting the lives of approximately 4,000 people mainly considering the vulnerability of women and girls. In line with the DRR concept, structures were established in both refugee and host community settings and with contingency plans in place on impeding disasters such as cyclone. Community mobilization, which included mock drills were part of preparedness for early warming and 104 groups were covered. It is also done orientation and capacity building of 02 Union disaster management committee and 03 Rohingya block committee at Camp level. Published October 2018, World Vision Bangladesh. 6
Cross cutting themes The project focused on cross cutting themes such as addressing gender and impacting in needs of genders in emergency, mainstreaming of disability training the Union Disaster Management Committee on Risk Reduction measures to address climatic challenges. Evaluation Target Audiences The assessment will focus on vulnerable HHs for both refugees and host communities 4. Evaluation Type Impact assessment 5. Evaluation Purpose and Objectives World Vision Bangladesh seeks the services of external individual consultant/firm to carry out an impact assessment to measure impact resulting from the implementation of BRCRP project, focusing mainly on successes, challenges, intended and unintended outcomes. In addressing the broad objective of the study, the evaluator should pay attention to the following key assessment questions/specific objectives which are to; Assess if the Project s achievements were in line with planned objectives and results. Assess if the realised project benefits of are likely to continue after WV Bangladesh and BRCRP support ends Describe any major failures and challenges of the project to date, explaining why they have occurred. Explain how the challenges were addressed and what should/could be done differently to increase the effectiveness of the project for similar interventions in the near future. In view of the end of project impact assessment, evaluation criteria for humanitarian emergencies, reference is therefore given to, OCED DAC, where the impact assessment seeks to answer the questions on each of the criterion selected Relevancy and appropriateness To assess to what extent the project activities and objectives were suited to the needs/priorities of the host communities and refugees. Furthermore, check if the project s objectives were consistent with priority needs of the government and more so to the strategic objectives of the organization. In addressing this, the assessor must address the following questions; 1. To what extent are the objectives consistent with priority needs for host and refugee communities? 2. Were the activities and outputs of the project in line with the overall goal and the attainment of its objectives? 3. Are the activities and outputs of the project aligned with the intended impacts and effects? 4. Did the proposed interventions suit the target beneficiaries and more so addressed their emergency needs and priorities? Effectiveness To assess the attainment level of the intervention s intended outcomes. Considering various activities, the assessor should address the following questions; 1. To what extent were the strengths and weaknesses of the project design and implementation processes identified and addressed? Published October 2018, World Vision Bangladesh. 7
2. How the food list choice provided did impacted on the choices made by beneficiaries? If so how? 3. Which benefits have been realised a result of agriculture inputs received, what could have been done differently to maximise the use of these input? 4. To what extent did the voucher token able to reduce negative coping mechanisms for benefitting households? Efficiency To assess whether the project was cost-efficient, achieved on time and whether they were implemented in the most efficient way compared to alternatives. 1. Has the project been cost-effective or would other approaches have led to the achieving of the same results at more reasonable costs? 2. Were transport and logistics appropriately planned to ensure timely distribution? 3. Were project outputs/outcomes achieved at a reasonable cost? If so how? Impact and Sustainability The positive and negative, primary and secondary long term effects produced development intervention, directly or indirectly, intended or unintended. In addressing this, the following questions should be addressed. 1. To what extent were the cash grants utilised by benefitting households? Is there any notable change resulting from receiving the cash grants? 2. Which typical food types were beneficiaries accessing by using voucher system? From the food types selected by beneficiaries was there dietary diversity? What were perceived benefits from food types accessed? 3. Which plans are in place for DRR? Is the evidence that both communities host and refugee to sustain risk reduction measures, if so how? 4. Is there any evidence of improved capacities resulting from micro business plan, DRR, IGAs on farm and nonfarm activities? 7. Limitations As the baseline was not done, this might limit comparison of the situation before and after especially for quantitative indicators 8. Authority and Responsibility 8.1 Team Members and Roles See appendix 1 8.2 Partners (i.e., who will be involved in the evaluation?) The programme partners expected to provide the technical support during the survey are Government of Bangladesh through the Coordination Camp Unit, NGOs both at host Community and Rohingya Refugee Camp level. 9. Team Advisors WV BRCRP will also provide an advisory role by providing technical support through M&E unit and National Office. Field staff will be available to ensure the expected results at the end of the exercise are achieved. 10. Logistics The Consultant will be expected to arrange his/her travel arrangement in collaboration with the WV BRCRP M&E team before and during the Impact Assessment. The activities will include amongst others; accommodation and transport arrangements, engagement of data collectors and clerks. Published October 2018, World Vision Bangladesh. 8
11. Products a. The Consultant will be expected to produce the following; b. Impact Assessment plan c. Tools for data collection d. Interview notes, transcripts e. Journals from the stakeholders f. Copies of raw data in the form of SPSS Dataset with a clearly labeled syntax g. Themes from the qualitative analysis h. Log(s) of data analysis processes i. Impact Assessment Report and a Power point presentation Note: Impact Assessment Report Draft Report. No more than 30 pages (without annexes) and should include photos and quotes highlighting significant change, including, but not limited to, sections on context, sampling, methodology, findings (including table showing achievements per indicator/ activity), analysis and recommendations. Data quality checks will be done by the Consultant. 13. Documents Project proposal Monitoring reports (DIP, OSDM, PDM reports, monthly report, beneficiary lists) WV BRCRP strategy Project budget Project Inception Beneficiaries Assessment Report Compendium of indicators 14. Lessons Learned Lessons learnt and best/promising practices should be captured on utilisation of IGAs, cash and grant support, voucher support, DRR etc - As IGAs were meant for income generating but implemented during the emergency. What went on well? What did not go on well? What could be done different for similar future projects - How did the training and input support developed their new and alternative livelihoods skills. What went on well? What did not go on well? What could be done different for similar future projects? - Provision of agricultural inputs for homestead gardening and training to meet their basic nutritional requirements. What went on well? What did not go on well? What could be done different for similar future projects - Provision of vouchers for vulnerable refugee households, to what extent was it used to meet their emergency needs. What went on well? What did not go on well? What could be done different for similar future projects - Provision of NFIs (solar light, umbrella utensils, cooking fuel, hygiene material. How did these address their lifesaving services? What went on well? What did not go on well? What could be done different for similar future projects? 15: Qualifications Minimum qualifications and experience The Consultant is expected to have the following minimum qualification and experience, First Degree/Advanced degree in Development/Disaster Management studies, Project Management, Economics, Statistics, or any other relevant Social Science discipline At least 5 years of professional experience in Relief and Development programs Evaluation/Impact Assessment Published October 2018, World Vision Bangladesh. 9
Extensive experience in Cash Based Programming, food security, infrastructure assessments, with substantive experience in conducting End line Surveys and lessons learnt documentation Experience working in developing countries and in building partnership with national disaster management agencies or national and local organizations Successful track record in advocacy, networking and inter-agency collaboration and management Strong organizational skills, with an ability to effectively manage multiple tasks simultaneously Ability to work both independently and as part of a diverse team Excellent communication skills, both written and oral, in English Excellent research skills both for qualitative and quantitative research purposes Proven ability to write and present complex emergency or development relatedissues for a non-technical audience Prior experience in conducting Final or Mid-term Impact Assessments for international donor funded projects is an asset but not a requirement. Published October 2018, World Vision Bangladesh. 10
15. Appendices Evaluation Phase Planning Team members Evaluation Team Role/Primary Task Review Impact Assessment design Review data collection tools Review the data collection and sampling methodologies Prepare the team on methods and data collection tools Ensure quality and compliance to standards in data collection Provide the LEAP report format Management Consultant Coordinate all Impact Assessment processes (planning, data collection, analysis and report writing) Technical team leader of the Impact Assessment team design of the Impact Assessment methodology Develop all necessary data collection tools. These tools will be reviewed by WV BCRCP team to ensure that they adequately capture all the necessary data before adoption Use the appropriate method to calculate the sample size and carry out sampling of the population Train enumerators, supervisors and Focus Group Discussion facilitators and oversee the pre-test Supervise the implementation of the field work Design the database for data entry Responsible for data entry and quality data management Review relevant secondary data Carry out data analysis as well as report writing Present the draft report to WV BRCRP team and incorporate the feedback and compile a final report Data collection, data analysis and sharing of findings to the stakeholders including the community Facilitate Community Mobilization Plan the field work Field Staff Office Organize the meetings with stakeholders Timelines 15 Nov Published October 2018, World Vision Bangladesh. 11