Surveys as a tool to improve aid in fragile states The Asia Foundation s Afghanistan experience
A fragile state environment Highly dynamic environment; uncertain &unstable Strained state-society relations; distrust & suspicion Multiple interests and actors at play; high costs of navigation Insufficient and unreliable information Weak local capabilities Weak physical and transportation infrastructure
Some ways that surveys can help Capture trends in a dynamic environment, if done over time Monitor state-society relations over time What matters? Public opinion vs. expert opinion Help capture key issues that are normally sensitive and deeply political Specific issues unique to the state can be studied in depth Can be a public good and used by multiple actors, where coordination risks and fund flows are high Better inform program design Better inform spending (targeting and resourcing) Test assumptions on key issues As a common baseline to track progress Build local capacity to provide reliable information
Varying objectives of TAF surveys Objective 1) Inform host government policymakers and local policy debates 2) Articulate/reflect key concerns of conflict-affected populations Survey Survey of Public Perceptions of the Mindanao Peace Process (2008) Local Economic Governance Index (EGI) in 5 countries Voter surveys in pre-election periods (Bangladesh, Cambodia, Indonesia, Pakistan, among others) Democracy & Conflict in Southern Thailand Survey of the Afghan People (2006-11) 3) Inform program design Public perceptions of police (Timor-Leste, Bangladesh, Sri Lanka, Indonesia) Leaders of Influence surveys (Maldives, Bangladesh) 4) Inform donor strategies or international policy Survey of the Afghan People (2006-11) Survey of the Sri Lankan People (2010) 5) Measure impact of programs Community policing pilot program in Sri Lanka, Timor-Leste (2009-11)
Case in point: Survey of the Afghan People Nationwide quantitative survey conducted annually using a structured questionnaire among more than 6200 Afghans 6 surveys conducted since baseline in 2006 Findings at http://www.asiafoundation.org/publications/pdf/989
Survey decision environment Pre survey During survey Post survey
Some Pre-survey Challenges Objective (s) of the survey Partnerships For what and with whom Localisation (Afghanisation) of the survey Ownership and accountability Objectivity Capacity
Some Challenges during Survey In general, minimal secondary information available for design In Afghanistan, last census in 1979, partial only No electoral rolls Questionnaire development Compatibility to other surveys within & outside the context Collaborative inputs Curiosity questions vs. actionable findings Scale of the fieldwork Weak to nonexistent local capacity Challenges with female interviewers
Some Challenges during Survey Weak infrastructure and harsh weather conditions Mountainous terrain & lack of (good) roads Long and harsh winters Ethno-linguistic sensitivities/closed society Conservatism Insularity Respondent selection and gender balance Balanced gender coverage at each sample point Travel restrictions for female interviewers
Some Challenges during Survey Quality control Added oversight and supervision of field team and data cleaning Backchecks and accompanied interviews Managing sampling and error Increasing insecurity and inaccessibility Greater replacement of sampling points Error margin can increase
Some Post-Survey and Other Challenges Analysis and reporting considerations Overstating the findings Interpreting every finding Emphasizing magnitude vs. direction Looking beyond topline findings to in-depth analysis, when to move along? When to integrate with comparisons to other worldwide surveys Dissemination plans and the ethics of research Selective data dissemination, suppression of findings Duplicative efforts, sharing of information
Some Post-Survey and Other Challenges Risk management Antigovernment elements, local strongmen, competitors Host Government Donors Way forward: one-off exercise vs. longitudinal time series Changing objective over time of conducting large scale surveys of Afghans Disciplined to team, timing, and, quality control costs Choice between parachute survey firm and in-house management in early years
Pictures from the field
Team training
The terrain
Interviews
Thank You!
Objectives of the Afghan Survey Use strong social science research to provide credible, policyrelevant information about public opinion on recent changes and trends in government performance, public policy, politics and political processes, and development progress Build social research capacity in the country
Dealing with outdated population data Rural sampling -- Combination of PPS and SRS Urban sampling -- use of maps/locality lists in the absence of database such as electoral rolls Innovative field movements for selection of starting points and households
Building the field team Extensive training of interviewers & supervisors Questionnaire Sampling procedure- field movements, selection of households Kish grid Train the trainer sessions Mock interviews Female interviewer recruitment in each district of selected sample point
Innovations Team of women interviewers recruited locally Mahrams for women interviewers Men and women interviewers in each sampling area Modifications in Kish selections Women interviewers obtained details of female HH members; men obtained details of males